tg-set-prompt
Sets prompt templates and system prompts for TrustGraph LLM services.
Synopsis
# Set a prompt template
tg-set-prompt --id TEMPLATE_ID --prompt TEMPLATE [options]
# Set system prompt
tg-set-prompt --system SYSTEM_PROMPT
Description
The tg-set-prompt command configures prompt templates and system prompts used by TrustGraph’s LLM services. Prompt templates contain placeholders like `` that are replaced with actual values when invoked. System prompts provide global context for all LLM interactions.
Templates are stored in TrustGraph’s configuration system and can be used with tg-invoke-prompt for consistent AI interactions.
Options
Prompt Template Mode
--id ID: Unique identifier for the prompt template (required for templates)--prompt TEMPLATE: Prompt template text with `` placeholders (required for templates)--response TYPE: Response format -textorjson(default:text)--schema SCHEMA: JSON schema for structured responses (required when response isjson)
System Prompt Mode
--system PROMPT: System prompt text (cannot be used with other options)
Common Options
-u, --api-url URL: TrustGraph API URL (default:$TRUSTGRAPH_URLorhttp://localhost:8088/)
Examples
Basic Prompt Template
tg-set-prompt \
--id "greeting" \
--prompt "Hello , welcome to !"
Question-Answer Template
tg-set-prompt \
--id "question" \
--prompt "Answer this question based on the context: \n\nContext: "
JSON Response Template
tg-set-prompt \
--id "extract-info" \
--prompt "Extract key information from: " \
--response "json" \
--schema '{"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "number"}}}'
Analysis Template
tg-set-prompt \
--id "analyze" \
--prompt "Analyze the following and provide insights about :\n\n\n\nFormat the response as ."
System Prompt
tg-set-prompt \
--system "You are a helpful AI assistant. Always provide accurate, concise responses. When uncertain, clearly state your limitations."
Template Variables
Variable Syntax
Templates use `` syntax for placeholders:
# Template
"Hello , today is "
# Usage
tg-invoke-prompt greeting name="Alice" day="Monday"
# Result: "Hello Alice, today is Monday"
Common Variables
- `` - Input text for processing
- `` - Question to answer
- `` - Background context
- `` - Data to analyze
- `` - Output format specification
Response Types
Text Response (Default)
tg-set-prompt \
--id "summarize" \
--prompt "Summarize this text in words: "
JSON Response
tg-set-prompt \
--id "classify" \
--prompt "Classify this text: " \
--response "json" \
--schema '{
"type": "object",
"properties": {
"category": {"type": "string"},
"confidence": {"type": "number", "minimum": 0, "maximum": 1}
},
"required": ["category", "confidence"]
}'
Use Cases
Document Processing Templates
# Document summarization
tg-set-prompt \
--id "document-summary" \
--prompt "Provide a summary of this document:\n\n\n\nFocus on: "
# Key point extraction
tg-set-prompt \
--id "extract-key-points" \
--prompt "Extract the main points from: \n\nReturn as a bulleted list."
# Document classification
tg-set-prompt \
--id "classify-document" \
--prompt "Classify this document into one of these categories: \n\nDocument: " \
--response "json" \
--schema '{"type": "object", "properties": {"category": {"type": "string"}, "confidence": {"type": "number"}}}'
Code Analysis Templates
# Code review
tg-set-prompt \
--id "code-review" \
--prompt "Review this code for issues:\n\n\n\nProvide specific recommendations."
# Bug detection
tg-set-prompt \
--id "find-bugs" \
--prompt "Analyze this code for potential bugs:\n\n\n\nError context: "
# Code explanation
tg-set-prompt \
--id "explain-code" \
--prompt "Explain how this code works:\n\n\n\nTarget audience: "
Data Analysis Templates
# Data insights
tg-set-prompt \
--id "data-insights" \
--prompt "Analyze this data and provide insights:\n\n\n\nFocus on: "
# Trend analysis
tg-set-prompt \
--id "trend-analysis" \
--prompt "Identify trends in this data over :\n\n" \
--response "json" \
--schema '{"type": "object", "properties": {"trends": {"type": "array", "items": {"type": "string"}}}}'
Content Generation Templates
# Marketing copy
tg-set-prompt \
--id "marketing-copy" \
--prompt "Create marketing copy for targeting . Key features: "
# Technical documentation
tg-set-prompt \
--id "tech-docs" \
--prompt "Generate technical documentation for:\n\n\n\nInclude: "
Advanced Usage
Multi-Step Templates
# Research template
tg-set-prompt \
--id "research" \
--prompt "Research question:
Available sources:
Please:
1. Analyze the question
2. Review relevant sources
3. Synthesize findings
4. Provide conclusions
Format: "
Conditional Templates
# Adaptive response template
tg-set-prompt \
--id "adaptive-response" \
--prompt "Task:
Context:
Expertise level:
If expertise level is 'beginner', provide simple explanations.
If expertise level is 'advanced', include technical details.
If task involves code, include examples.
Response:"
Structured Analysis Template
tg-set-prompt \
--id "structured-analysis" \
--prompt "Analyze:
Criteria:
Data:
Provide analysis in this structure:
- Overview
- Key Findings
- Recommendations
- Next Steps" \
--response "json" \
--schema '{
"type": "object",
"properties": {
"overview": {"type": "string"},
"key_findings": {"type": "array", "items": {"type": "string"}},
"recommendations": {"type": "array", "items": {"type": "string"}},
"next_steps": {"type": "array", "items": {"type": "string"}}
}
}'
Template Management
# Create template collection for specific domain
domain="customer-support"
templates=(
"greeting:Hello! I'm here to help with . What can I assist you with?"
"escalation:I understand your frustration with . Let me escalate this to ."
"resolution:Great! I've resolved your . Is there anything else I can help with?"
)
for template in "${templates[@]}"; do
IFS=':' read -r id prompt <<< "$template"
tg-set-prompt --id "${domain}-${id}" --prompt "$prompt"
done
System Prompt Configuration
General Purpose System Prompt
tg-set-prompt --system "You are a knowledgeable AI assistant. Provide accurate, helpful responses. When you don't know something, say so clearly. Always consider the context and be concise unless detail is specifically requested."
Domain-Specific System Prompt
tg-set-prompt --system "You are a technical documentation assistant specializing in software development. Focus on clarity, accuracy, and practical examples. Always include code snippets when relevant and explain complex concepts step-by-step."
Role-Based System Prompt
tg-set-prompt --system "You are a data analyst AI. When analyzing data, always consider statistical significance, potential biases, and limitations. Present findings objectively and suggest actionable insights."
Error Handling
Missing Required Fields
Exception: Must specify --id for prompt
Solution: Provide both --id and --prompt for template creation.
Invalid Response Type
Exception: Response must be one of: text json
Solution: Use only text or json for the --response option.
Invalid JSON Schema
Exception: JSON schema must be valid JSON
Solution: Validate JSON schema syntax before using --schema.
Conflicting Options
Exception: Can't use --system with other args
Solution: Use --system alone, or use template options without --system.
Template Testing
Test Template Creation
# Create and test a simple template
tg-set-prompt \
--id "test-template" \
--prompt "Test template with and "
# Test the template
tg-invoke-prompt test-template variable1="hello" variable2="world"
Validate JSON Templates
# Create JSON template
tg-set-prompt \
--id "json-test" \
--prompt "Extract data from: " \
--response "json" \
--schema '{"type": "object", "properties": {"result": {"type": "string"}}}'
# Test JSON response
tg-invoke-prompt json-test text="Sample text for testing"
Template Iteration
# Version 1
tg-set-prompt \
--id "analysis-v1" \
--prompt "Analyze: "
# Version 2 (improved)
tg-set-prompt \
--id "analysis-v2" \
--prompt "Analyze the following and provide insights about :\n\n\n\nConsider: "
# Version 3 (structured)
tg-set-prompt \
--id "analysis-v3" \
--prompt "Analyze: " \
--response "json" \
--schema '{"type": "object", "properties": {"summary": {"type": "string"}, "insights": {"type": "array"}}}'
Best Practices
Template Design
# Good: Clear, specific prompts
tg-set-prompt \
--id "good-summary" \
--prompt "Summarize this in words, focusing on :\n\n
<!DOCTYPE html>
<html lang="en-US">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=Edge">
<link rel="stylesheet" href="/assets/css/just-the-docs-default.css">
<link rel="stylesheet" href="/assets/css/just-the-docs-head-nav.css" id="jtd-head-nav-stylesheet">
<style id="jtd-nav-activation">
.site-nav > ul.nav-list:first-child > li > a,
.site-nav > ul.nav-list:first-child > li > ul > li > a,
.site-nav > ul.nav-list:first-child > li > ul > li > ul > li > a,
.site-nav > ul.nav-list:first-child > li > ul > li > ul > li > ul > li:not(:nth-child(24)) > a,
.site-nav > ul.nav-list:first-child > li > ul > li > ul > li > ul > li > ul > li a {
background-image: none;
}
.site-nav > ul.nav-list:not(:first-child) a,
.site-nav li.external a {
background-image: none;
}
.site-nav > ul.nav-list:first-child > li:nth-child(1) > ul > li:nth-child(5) > ul > li:nth-child(1) > ul > li:nth-child(24) > a {
font-weight: 600;
text-decoration: none;
}.site-nav > ul.nav-list:first-child > li:nth-child(1) > button svg,
.site-nav > ul.nav-list:first-child > li:nth-child(1) > ul > li:nth-child(5) > button svg,
.site-nav > ul.nav-list:first-child > li:nth-child(1) > ul > li:nth-child(5) > ul > li:nth-child(1) > button svg,
.site-nav > ul.nav-list:first-child > li:nth-child(1) > ul > li:nth-child(5) > ul > li:nth-child(1) > ul > li:nth-child(24) > button svg {
transform: rotate(-90deg);
}.site-nav > ul.nav-list:first-child > li.nav-list-item:nth-child(1) > ul.nav-list,
.site-nav > ul.nav-list:first-child > li.nav-list-item:nth-child(1) > ul.nav-list > li.nav-list-item:nth-child(5) > ul.nav-list,
.site-nav > ul.nav-list:first-child > li.nav-list-item:nth-child(1) > ul.nav-list > li.nav-list-item:nth-child(5) > ul.nav-list > li.nav-list-item:nth-child(1) > ul.nav-list,
.site-nav > ul.nav-list:first-child > li.nav-list-item:nth-child(1) > ul.nav-list > li.nav-list-item:nth-child(5) > ul.nav-list > li.nav-list-item:nth-child(1) > ul.nav-list > li.nav-list-item:nth-child(24) > ul.nav-list {
display: block;
}
</style>
<script src="/assets/js/vendor/lunr.min.js"></script>
<script src="/assets/js/just-the-docs.js"></script>
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- Begin Jekyll SEO tag v2.8.0 -->
<title>tg-save-doc-embeds | TrustGraph</title>
<meta name="generator" content="Jekyll v4.4.1" />
<meta property="og:title" content="tg-save-doc-embeds" />
<meta property="og:locale" content="en_US" />
<meta name="description" content="Information on the TrustGraph platform" />
<meta property="og:description" content="Information on the TrustGraph platform" />
<link rel="canonical" href="https://just-the-docs.github.io/reference/cli/tg-save-doc-embeds.html" />
<meta property="og:url" content="https://just-the-docs.github.io/reference/cli/tg-save-doc-embeds.html" />
<meta property="og:site_name" content="TrustGraph" />
<meta property="og:type" content="website" />
<meta name="twitter:card" content="summary" />
<meta property="twitter:title" content="tg-save-doc-embeds" />
<script type="application/ld+json">
{"@context":"https://schema.org","@type":"WebPage","description":"Information on the TrustGraph platform","headline":"tg-save-doc-embeds","url":"https://just-the-docs.github.io/reference/cli/tg-save-doc-embeds.html"}</script>
<!-- End Jekyll SEO tag -->
</head>
<body>
<a class="skip-to-main" href="#main-content">Skip to main content</a>
<svg xmlns="http://www.w3.org/2000/svg" class="d-none">
<symbol id="svg-link" viewBox="0 0 24 24">
<title>Link</title>
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-link">
<path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"></path><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"></path>
</svg>
</symbol>
<symbol id="svg-menu" viewBox="0 0 24 24">
<title>Menu</title>
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-menu">
<line x1="3" y1="12" x2="21" y2="12"></line><line x1="3" y1="6" x2="21" y2="6"></line><line x1="3" y1="18" x2="21" y2="18"></line>
</svg>
</symbol>
<symbol id="svg-arrow-right" viewBox="0 0 24 24">
<title>Expand</title>
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-chevron-right">
<polyline points="9 18 15 12 9 6"></polyline>
</svg>
</symbol>
<!-- Feather. MIT License: https://github.com/feathericons/feather/blob/master/LICENSE -->
<symbol id="svg-external-link" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-external-link">
<title id="svg-external-link-title">(external link)</title>
<path d="M18 13v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"></path><polyline points="15 3 21 3 21 9"></polyline><line x1="10" y1="14" x2="21" y2="3"></line>
</symbol>
<symbol id="svg-doc" viewBox="0 0 24 24">
<title>Document</title>
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-file">
<path d="M13 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V9z"></path><polyline points="13 2 13 9 20 9"></polyline>
</svg>
</symbol>
<symbol id="svg-search" viewBox="0 0 24 24">
<title>Search</title>
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-search">
<circle cx="11" cy="11" r="8"></circle><line x1="21" y1="21" x2="16.65" y2="16.65"></line>
</svg>
</symbol>
<!-- Bootstrap Icons. MIT License: https://github.com/twbs/icons/blob/main/LICENSE.md -->
<symbol id="svg-copy" viewBox="0 0 16 16">
<title>Copy</title>
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentColor" class="bi bi-clipboard" viewBox="0 0 16 16">
<path d="M4 1.5H3a2 2 0 0 0-2 2V14a2 2 0 0 0 2 2h10a2 2 0 0 0 2-2V3.5a2 2 0 0 0-2-2h-1v1h1a1 1 0 0 1 1 1V14a1 1 0 0 1-1 1H3a1 1 0 0 1-1-1V3.5a1 1 0 0 1 1-1h1v-1z"/>
<path d="M9.5 1a.5.5 0 0 1 .5.5v1a.5.5 0 0 1-.5.5h-3a.5.5 0 0 1-.5-.5v-1a.5.5 0 0 1 .5-.5h3zm-3-1A1.5 1.5 0 0 0 5 1.5v1A1.5 1.5 0 0 0 6.5 4h3A1.5 1.5 0 0 0 11 2.5v-1A1.5 1.5 0 0 0 9.5 0h-3z"/>
</svg>
</symbol>
<symbol id="svg-copied" viewBox="0 0 16 16">
<title>Copied</title>
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentColor" class="bi bi-clipboard-check-fill" viewBox="0 0 16 16">
<path d="M6.5 0A1.5 1.5 0 0 0 5 1.5v1A1.5 1.5 0 0 0 6.5 4h3A1.5 1.5 0 0 0 11 2.5v-1A1.5 1.5 0 0 0 9.5 0h-3Zm3 1a.5.5 0 0 1 .5.5v1a.5.5 0 0 1-.5.5h-3a.5.5 0 0 1-.5-.5v-1a.5.5 0 0 1 .5-.5h3Z"/>
<path d="M4 1.5H3a2 2 0 0 0-2 2V14a2 2 0 0 0 2 2h10a2 2 0 0 0 2-2V3.5a2 2 0 0 0-2-2h-1v1A2.5 2.5 0 0 1 9.5 5h-3A2.5 2.5 0 0 1 4 2.5v-1Zm6.854 7.354-3 3a.5.5 0 0 1-.708 0l-1.5-1.5a.5.5 0 0 1 .708-.708L7.5 10.793l2.646-2.647a.5.5 0 0 1 .708.708Z"/>
</svg>
</symbol>
</svg>
<div class="side-bar">
<div class="site-header" role="banner">
<a href="/" class="site-title lh-tight">
TrustGraph
</a>
<button id="menu-button" class="site-button btn-reset" aria-label="Toggle menu" aria-pressed="false">
<svg viewBox="0 0 24 24" class="icon" aria-hidden="true"><use xlink:href="#svg-menu"></use></svg>
</button>
</div>
<nav aria-label="Main" id="site-nav" class="site-nav">
<ul class="nav-list"><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in TrustGraph Documentation category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/" class="nav-list-link">TrustGraph Documentation</a><ul class="nav-list"><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Getting Started category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/getting-started/" class="nav-list-link">Getting Started</a><ul class="nav-list"><li class="nav-list-item"><a href="/getting-started/concepts.html" class="nav-list-link">Core Concepts</a></li><li class="nav-list-item"><a href="/getting-started/installation.html" class="nav-list-link">Installation</a></li><li class="nav-list-item"><a href="/getting-started/first-steps.html" class="nav-list-link">First Steps</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Overview category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/overview/" class="nav-list-link">Overview</a><ul class="nav-list"><li class="nav-list-item"><a href="/overview/philosophy.html" class="nav-list-link">Philosophy</a></li><li class="nav-list-item"><a href="/overview/architecture.html" class="nav-list-link">Architecture</a></li><li class="nav-list-item"><a href="/overview/features.html" class="nav-list-link">Features</a></li><li class="nav-list-item"><a href="/overview/use-cases.html" class="nav-list-link">Use Cases</a></li><li class="nav-list-item"><a href="/overview/feature-maturity.html" class="nav-list-link">Feature Maturity</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Deployment category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/deployment/" class="nav-list-link">Deployment</a><ul class="nav-list"><li class="nav-list-item"><a href="/deployment/docker-compose.html" class="nav-list-link">Docker Compose / Podman Compose</a></li><li class="nav-list-item"><a href="/deployment/minikube.html" class="nav-list-link">Minikube</a></li><li class="nav-list-item"><a href="/deployment/intel.html" class="nav-list-link">Intel GPU / Tiber Cloud</a></li><li class="nav-list-item"><a href="/deployment/scaleway.html" class="nav-list-link">Scaleway</a></li><li class="nav-list-item"><a href="/deployment/azure.html" class="nav-list-link">Azure AKS</a></li><li class="nav-list-item"><a href="/deployment/gcp.html" class="nav-list-link">Google Cloud Platform</a></li><li class="nav-list-item"><a href="/deployment/aws-ec2.html" class="nav-list-link">AWS EC2 Single Instance</a></li><li class="nav-list-item"><a href="/deployment/aws-rke.html" class="nav-list-link">Amazon Web Services (RKE)</a></li><li class="nav-list-item"><a href="/deployment/security-considerations.html" class="nav-list-link">Security Considerations</a></li><li class="nav-list-item"><a href="/deployment/production-considerations.html" class="nav-list-link">Production Considerations</a></li><li class="nav-list-item"><a href="/deployment/troubleshooting.html" class="nav-list-link">Troubleshooting</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in How-to Guides category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/guides/" class="nav-list-link">How-to Guides</a><ul class="nav-list"><li class="nav-list-item"><a href="/guides/data-integration/" class="nav-list-link">Data Integration</a></li><li class="nav-list-item"><a href="/guides/mcp-integration/" class="nav-list-link">MCP Integration</a></li><li class="nav-list-item"><a href="/guides/migration/" class="nav-list-link">Migration</a></li><li class="nav-list-item"><a href="/guides/monitoring/" class="nav-list-link">Monitoring</a></li><li class="nav-list-item"><a href="/guides/querying/" class="nav-list-link">Querying</a></li><li class="nav-list-item"><a href="/guides/security/" class="nav-list-link">Security</a></li><li class="nav-list-item"><a href="/guides/visualization/" class="nav-list-link">Visualization</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Reference category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/reference/" class="nav-list-link">Reference</a><ul class="nav-list"><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in CLI category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/reference/cli/" class="nav-list-link">CLI</a><ul class="nav-list"><li class="nav-list-item"><a href="/reference/cli/tg-add-library-document.html" class="nav-list-link">tg-add-library-document</a></li><li class="nav-list-item"><a href="/reference/cli/tg-delete-flow-class.html" class="nav-list-link">tg-delete-flow-class</a></li><li class="nav-list-item"><a href="/reference/cli/tg-delete-kg-core.html" class="nav-list-link">tg-delete-kg-core</a></li><li class="nav-list-item"><a href="/reference/cli/tg-dump-msgpack.html" class="nav-list-link">tg-dump-msgpack</a></li><li class="nav-list-item"><a href="/reference/cli/tg-get-flow-class.html" class="nav-list-link">tg-get-flow-class</a></li><li class="nav-list-item"><a href="/reference/cli/tg-get-kg-core.html" class="nav-list-link">tg-get-kg-core</a></li><li class="nav-list-item"><a href="/reference/cli/tg-graph-to-turtle.html" class="nav-list-link">tg-graph-to-turtle</a></li><li class="nav-list-item"><a href="/reference/cli/tg-init-pulsar-manager.html" class="nav-list-link">tg-init-pulsar-manager</a></li><li class="nav-list-item"><a href="/reference/cli/tg-init-trustgraph.html" class="nav-list-link">tg-init-trustgraph</a></li><li class="nav-list-item"><a href="/reference/cli/tg-invoke-agent.html" class="nav-list-link">tg-invoke-agent</a></li><li class="nav-list-item"><a href="/reference/cli/tg-invoke-document-rag.html" class="nav-list-link">tg-invoke-document-rag</a></li><li class="nav-list-item"><a href="/reference/cli/tg-invoke-graph-rag.html" class="nav-list-link">tg-invoke-graph-rag</a></li><li class="nav-list-item"><a href="/reference/cli/tg-invoke-llm.html" class="nav-list-link">tg-invoke-llm</a></li><li class="nav-list-item"><a href="/reference/cli/tg-invoke-prompt.html" class="nav-list-link">tg-invoke-prompt</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-doc-embeds.html" class="nav-list-link">tg-load-doc-embeds</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-kg-core.html" class="nav-list-link">tg-load-kg-core</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-pdf.html" class="nav-list-link">tg-load-pdf</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-sample-documents.html" class="nav-list-link">tg-load-sample-documents</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-text.html" class="nav-list-link">tg-load-text</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-turtle.html" class="nav-list-link">tg-load-turtle</a></li><li class="nav-list-item"><a href="/reference/cli/tg-put-flow-class.html" class="nav-list-link">tg-put-flow-class</a></li><li class="nav-list-item"><a href="/reference/cli/tg-put-kg-core.html" class="nav-list-link">tg-put-kg-core</a></li><li class="nav-list-item"><a href="/reference/cli/tg-remove-library-document.html" class="nav-list-link">tg-remove-library-document</a></li><li class="nav-list-item"><a href="/reference/cli/tg-save-doc-embeds.html" class="nav-list-link">tg-save-doc-embeds</a></li><li class="nav-list-item"><a href="/reference/cli/tg-set-prompt.html" class="nav-list-link">tg-set-prompt</a></li><li class="nav-list-item"><a href="/reference/cli/tg-set-token-costs.html" class="nav-list-link">tg-set-token-costs</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-config.html" class="nav-list-link">tg-show-config</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-flow-classes.html" class="nav-list-link">tg-show-flow-classes</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-flow-state.html" class="nav-list-link">tg-show-flow-state</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-flows.html" class="nav-list-link">tg-show-flows</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-graph.html" class="nav-list-link">tg-show-graph</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-kg-cores.html" class="nav-list-link">tg-show-kg-cores</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-library-documents.html" class="nav-list-link">tg-show-library-documents</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-library-processing.html" class="nav-list-link">tg-show-library-processing</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-processor-state.html" class="nav-list-link">tg-show-processor-state</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-prompts.html" class="nav-list-link">tg-show-prompts</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-token-costs.html" class="nav-list-link">tg-show-token-costs</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-token-rate.html" class="nav-list-link">tg-show-token-rate</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-tools.html" class="nav-list-link">tg-show-tools</a></li><li class="nav-list-item"><a href="/reference/cli/tg-start-flow.html" class="nav-list-link">tg-start-flow</a></li><li class="nav-list-item"><a href="/reference/cli/tg-start-library-processing.html" class="nav-list-link">tg-start-library-processing</a></li><li class="nav-list-item"><a href="/reference/cli/tg-stop-flow.html" class="nav-list-link">tg-stop-flow</a></li><li class="nav-list-item"><a href="/reference/cli/tg-stop-library-processing.html" class="nav-list-link">tg-stop-library-processing</a></li><li class="nav-list-item"><a href="/reference/cli/tg-unload-kg-core.html" class="nav-list-link">tg-unload-kg-core</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in APIs category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/reference/apis/" class="nav-list-link">APIs</a><ul class="nav-list"><li class="nav-list-item"><a href="/reference/apis/pulsar.html" class="nav-list-link">About Pulsar</a></li><li class="nav-list-item"><a href="/reference/apis/api-agent.html" class="nav-list-link">Agent API</a></li><li class="nav-list-item"><a href="/reference/apis/api-config.html" class="nav-list-link">Config API</a></li><li class="nav-list-item"><a href="/reference/apis/api-core-import-export.html" class="nav-list-link">Core Import/Export API</a></li><li class="nav-list-item"><a href="/reference/apis/api-document-embeddings.html" class="nav-list-link">Document Embeddings API</a></li><li class="nav-list-item"><a href="/reference/apis/api-document-load.html" class="nav-list-link">Document Load API</a></li><li class="nav-list-item"><a href="/reference/apis/api-document-rag.html" class="nav-list-link">Document RAG API</a></li><li class="nav-list-item"><a href="/reference/apis/api-embeddings.html" class="nav-list-link">Embeddings API</a></li><li class="nav-list-item"><a href="/reference/apis/api-entity-contexts.html" class="nav-list-link">Entity Contexts API</a></li><li class="nav-list-item"><a href="/reference/apis/api-flow.html" class="nav-list-link">Flow API</a></li><li class="nav-list-item"><a href="/reference/apis/api-graph-embeddings.html" class="nav-list-link">Graph Embeddings API</a></li><li class="nav-list-item"><a href="/reference/apis/api-graph-rag.html" class="nav-list-link">Graph RAG API</a></li><li class="nav-list-item"><a href="/reference/apis/api-knowledge.html" class="nav-list-link">Knowledge API</a></li><li class="nav-list-item"><a href="/reference/apis/api-librarian.html" class="nav-list-link">Librarian API</a></li><li class="nav-list-item"><a href="/reference/apis/api-metrics.html" class="nav-list-link">Metrics API</a></li><li class="nav-list-item"><a href="/reference/apis/api-prompt.html" class="nav-list-link">Prompt API</a></li><li class="nav-list-item"><a href="/reference/apis/api-text-completion.html" class="nav-list-link">Text Completion API</a></li><li class="nav-list-item"><a href="/reference/apis/api-text-load.html" class="nav-list-link">Text Load API</a></li><li class="nav-list-item"><a href="/reference/apis/api-triples-query.html" class="nav-list-link">Triples Query API</a></li><li class="nav-list-item"><a href="/reference/apis/websocket.html" class="nav-list-link">websocket</a></li></ul></li><li class="nav-list-item"><a href="/reference/containers.html" class="nav-list-link">Containers</a></li><li class="nav-list-item"><a href="/reference/python-packages.html" class="nav-list-link">Python packages</a></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Extending TrustGraph category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/reference/extending/" class="nav-list-link">Extending TrustGraph</a><ul class="nav-list"><li class="nav-list-item"><a href="/reference/extending/async-processor.html" class="nav-list-link">AsyncProcessor</a></li><li class="nav-list-item"><a href="/reference/extending/flow-processor.html" class="nav-list-link">FlowProcessor</a></li><li class="nav-list-item"><a href="/reference/extending/service-base-classes.html" class="nav-list-link">Service Base Classes</a></li><li class="nav-list-item"><a href="/reference/extending/flow-specifications.html" class="nav-list-link">Flow Specifications</a></li></ul></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Community category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/community/" class="nav-list-link">Community</a><ul class="nav-list"><li class="nav-list-item"><a href="/community/changelog.html" class="nav-list-link">Changelog</a></li><li class="nav-list-item"><a href="/community/code-of-conduct.html" class="nav-list-link">Code of Conduct</a></li><li class="nav-list-item"><a href="/community/contributing.html" class="nav-list-link">Contributing</a></li><li class="nav-list-item"><a href="/community/developer.html" class="nav-list-link">Developer's Guide</a></li><li class="nav-list-item"><a href="/community/roadmap.html" class="nav-list-link">Roadmap</a></li><li class="nav-list-item"><a href="/community/support.html" class="nav-list-link">Support</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Examples category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/examples/" class="nav-list-link">Examples</a><ul class="nav-list"><li class="nav-list-item"><a href="/examples/integrations/" class="nav-list-link">Integrations</a></li><li class="nav-list-item"><a href="/examples/sample-data/" class="nav-list-link">Sample Data</a></li><li class="nav-list-item"><a href="/examples/tutorials/" class="nav-list-link">Tutorials</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Advanced Topics category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/advanced/" class="nav-list-link">Advanced Topics</a><ul class="nav-list"><li class="nav-list-item"><a href="/advanced/backup-restore.html" class="nav-list-link">Backup & Restore</a></li><li class="nav-list-item"><a href="/advanced/clustering.html" class="nav-list-link">Clustering</a></li><li class="nav-list-item"><a href="/advanced/custom-algorithms.html" class="nav-list-link">Custom Algorithms</a></li><li class="nav-list-item"><a href="/advanced/disaster-recovery.html" class="nav-list-link">Disaster Recovery</a></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Extending TrustGraph category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/advanced/extending-trustgraph.html" class="nav-list-link">Extending TrustGraph</a><ul class="nav-list"><li class="nav-list-item"><a href="/reference/extending/async-processor.html" class="nav-list-link">AsyncProcessor</a></li><li class="nav-list-item"><a href="/reference/extending/flow-processor.html" class="nav-list-link">FlowProcessor</a></li><li class="nav-list-item"><a href="/reference/extending/service-base-classes.html" class="nav-list-link">Service Base Classes</a></li><li class="nav-list-item"><a href="/reference/extending/flow-specifications.html" class="nav-list-link">Flow Specifications</a></li></ul></li><li class="nav-list-item"><a href="/advanced/performance-tuning.html" class="nav-list-link">Performance Tuning</a></li></ul></li></ul></li></ul>
</nav>
<footer class="site-footer">
This site uses <a href="https://github.com/just-the-docs/just-the-docs">Just the Docs</a>, a documentation theme for Jekyll.
</footer>
</div>
<div class="main" id="top">
<div id="main-header" class="main-header">
<div class="search" role="search">
<div class="search-input-wrap">
<input type="text" id="search-input" class="search-input" tabindex="0" placeholder="Search TrustGraph" aria-label="Search TrustGraph" autocomplete="off">
<label for="search-input" class="search-label"><svg viewBox="0 0 24 24" class="search-icon"><use xlink:href="#svg-search"></use></svg></label>
</div>
<div id="search-results" class="search-results"></div>
</div>
<nav aria-label="Auxiliary" class="aux-nav">
<ul class="aux-nav-list">
<li class="aux-nav-list-item">
<a href="//github.com/trustgraph-ai/trustgraph" class="site-button"
target="_blank" rel="noopener noreferrer"
>
TrustGraph on GitHub
</a>
</li>
</ul>
</nav>
</div>
<div class="main-content-wrap">
<nav aria-label="Breadcrumb" class="breadcrumb-nav">
<ol class="breadcrumb-nav-list">
<li class="breadcrumb-nav-list-item"><a href="/">TrustGraph Documentation</a></li>
<li class="breadcrumb-nav-list-item"><a href="/reference/">Reference</a></li>
<li class="breadcrumb-nav-list-item"><a href="/reference/cli/">CLI</a></li>
<li class="breadcrumb-nav-list-item"><span>tg-save-doc-embeds</span></li>
</ol>
</nav>
<div id="main-content" class="main-content">
<main>
<h1 id="tg-save-doc-embeds">
<a href="#tg-save-doc-embeds" class="anchor-heading" aria-labelledby="tg-save-doc-embeds"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> tg-save-doc-embeds
</h1>
<p>Saves document embeddings from TrustGraph processing streams to MessagePack format files.</p>
<h2 id="synopsis">
<a href="#synopsis" class="anchor-heading" aria-labelledby="synopsis"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Synopsis
</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="nt">-o</span> OUTPUT_FILE <span class="o">[</span>options]
</code></pre></div></div>
<h2 id="description">
<a href="#description" class="anchor-heading" aria-labelledby="description"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Description
</h2>
<p>The <code class="language-plaintext highlighter-rouge">tg-save-doc-embeds</code> command connects to TrustGraph’s document embeddings export stream and saves the embeddings to a file in MessagePack format. This is useful for creating backups of document embeddings, exporting data for analysis, or preparing data for migration between systems.</p>
<p>The command should typically be started before document processing begins to capture all embeddings as they are generated.</p>
<h2 id="options">
<a href="#options" class="anchor-heading" aria-labelledby="options"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Options
</h2>
<h3 id="required-arguments">
<a href="#required-arguments" class="anchor-heading" aria-labelledby="required-arguments"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Required Arguments
</h3>
<ul>
<li><code class="language-plaintext highlighter-rouge">-o, --output-file FILE</code>: Output file for saved embeddings</li>
</ul>
<h3 id="optional-arguments">
<a href="#optional-arguments" class="anchor-heading" aria-labelledby="optional-arguments"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Optional Arguments
</h3>
<ul>
<li><code class="language-plaintext highlighter-rouge">-u, --url URL</code>: TrustGraph API URL (default: <code class="language-plaintext highlighter-rouge">$TRUSTGRAPH_API</code> or <code class="language-plaintext highlighter-rouge">http://localhost:8088/</code>)</li>
<li><code class="language-plaintext highlighter-rouge">-f, --flow-id ID</code>: Flow instance ID to monitor (default: <code class="language-plaintext highlighter-rouge">default</code>)</li>
<li><code class="language-plaintext highlighter-rouge">--format FORMAT</code>: Output format - <code class="language-plaintext highlighter-rouge">msgpack</code> or <code class="language-plaintext highlighter-rouge">json</code> (default: <code class="language-plaintext highlighter-rouge">msgpack</code>)</li>
<li><code class="language-plaintext highlighter-rouge">--user USER</code>: Filter by user ID (default: no filter)</li>
<li><code class="language-plaintext highlighter-rouge">--collection COLLECTION</code>: Filter by collection ID (default: no filter)</li>
</ul>
<h2 id="examples">
<a href="#examples" class="anchor-heading" aria-labelledby="examples"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Examples
</h2>
<h3 id="basic-document-embeddings-export">
<a href="#basic-document-embeddings-export" class="anchor-heading" aria-labelledby="basic-document-embeddings-export"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Basic Document Embeddings Export
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="nt">-o</span> document-embeddings.msgpack
</code></pre></div></div>
<h3 id="export-from-specific-flow">
<a href="#export-from-specific-flow" class="anchor-heading" aria-labelledby="export-from-specific-flow"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Export from Specific Flow
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> research-embeddings.msgpack <span class="se">\</span>
<span class="nt">-f</span> <span class="s2">"research-processing-flow"</span>
</code></pre></div></div>
<h3 id="filter-by-user-and-collection">
<a href="#filter-by-user-and-collection" class="anchor-heading" aria-labelledby="filter-by-user-and-collection"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Filter by User and Collection
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> filtered-embeddings.msgpack <span class="se">\</span>
<span class="nt">--user</span> <span class="s2">"research-team"</span> <span class="se">\</span>
<span class="nt">--collection</span> <span class="s2">"research-docs"</span>
</code></pre></div></div>
<h3 id="export-to-json-format">
<a href="#export-to-json-format" class="anchor-heading" aria-labelledby="export-to-json-format"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Export to JSON Format
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> embeddings.json <span class="se">\</span>
<span class="nt">--format</span> json
</code></pre></div></div>
<h3 id="production-backup">
<a href="#production-backup" class="anchor-heading" aria-labelledby="production-backup"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Production Backup
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"backup-</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d-%H%M%S<span class="si">)</span><span class="s2">.msgpack"</span> <span class="se">\</span>
<span class="nt">-u</span> https://production-api.company.com/ <span class="se">\</span>
<span class="nt">-f</span> <span class="s2">"production-flow"</span>
</code></pre></div></div>
<h2 id="output-format">
<a href="#output-format" class="anchor-heading" aria-labelledby="output-format"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Output Format
</h2>
<h3 id="messagepack-structure">
<a href="#messagepack-structure" class="anchor-heading" aria-labelledby="messagepack-structure"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> MessagePack Structure
</h3>
<p>Document embeddings are saved as MessagePack records:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="s2">"de"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"m"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"i"</span><span class="p">:</span><span class="w"> </span><span class="s2">"document-id"</span><span class="p">,</span><span class="w">
</span><span class="nl">"m"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="nl">"metadata"</span><span class="p">:</span><span class="w"> </span><span class="s2">"objects"</span><span class="p">}],</span><span class="w">
</span><span class="nl">"u"</span><span class="p">:</span><span class="w"> </span><span class="s2">"user-id"</span><span class="p">,</span><span class="w">
</span><span class="nl">"c"</span><span class="p">:</span><span class="w"> </span><span class="s2">"collection-id"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"c"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"c"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text chunk content"</span><span class="p">,</span><span class="w">
</span><span class="nl">"v"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="mf">0.1</span><span class="p">,</span><span class="w"> </span><span class="mf">0.2</span><span class="p">,</span><span class="w"> </span><span class="mf">0.3</span><span class="p">,</span><span class="w"> </span><span class="err">...</span><span class="p">]</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span></code></pre></div></div>
<h3 id="components">
<a href="#components" class="anchor-heading" aria-labelledby="components"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Components
</h3>
<ul>
<li><strong>Record Type</strong>: <code class="language-plaintext highlighter-rouge">"de"</code> indicates document embeddings</li>
<li><strong>Metadata</strong> (<code class="language-plaintext highlighter-rouge">m</code>): Document information and context</li>
<li><strong>Chunks</strong> (<code class="language-plaintext highlighter-rouge">c</code>): Text chunks with their vector embeddings</li>
</ul>
<h2 id="use-cases">
<a href="#use-cases" class="anchor-heading" aria-labelledby="use-cases"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Use Cases
</h2>
<h3 id="backup-creation">
<a href="#backup-creation" class="anchor-heading" aria-labelledby="backup-creation"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Backup Creation
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Create regular backups of document embeddings</span>
create_embeddings_backup<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">backup_dir</span><span class="o">=</span><span class="s2">"embeddings-backups"</span>
<span class="nb">local </span><span class="nv">timestamp</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d_%H%M%S<span class="si">)</span>
<span class="nb">local </span><span class="nv">backup_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$backup_dir</span><span class="s2">/embeddings-</span><span class="nv">$timestamp</span><span class="s2">.msgpack"</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> <span class="s2">"</span><span class="nv">$backup_dir</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Creating embeddings backup: </span><span class="nv">$backup_file</span><span class="s2">"</span>
<span class="c"># Start backup process</span>
tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"</span><span class="nv">$backup_file</span><span class="s2">"</span> &
<span class="nv">save_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="nb">echo</span> <span class="s2">"Backup process started (PID: </span><span class="nv">$save_pid</span><span class="s2">)"</span>
<span class="nb">echo</span> <span class="s2">"To stop: kill </span><span class="nv">$save_pid</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Backup file: </span><span class="nv">$backup_file</span><span class="s2">"</span>
<span class="c"># Optionally wait for a specific duration</span>
<span class="c"># sleep 3600 # Run for 1 hour</span>
<span class="c"># kill $save_pid</span>
<span class="o">}</span>
<span class="c"># Create backup</span>
create_embeddings_backup
</code></pre></div></div>
<h3 id="data-migration-preparation">
<a href="#data-migration-preparation" class="anchor-heading" aria-labelledby="data-migration-preparation"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Data Migration Preparation
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Prepare embeddings for migration</span>
prepare_migration_data<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">source_env</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">collection</span><span class="o">=</span><span class="s2">"</span><span class="nv">$2</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">migration_file</span><span class="o">=</span><span class="s2">"migration-</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d<span class="si">)</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Preparing migration data from: </span><span class="nv">$source_env</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Collection: </span><span class="nv">$collection</span><span class="s2">"</span>
<span class="c"># Export embeddings from source</span>
tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"</span><span class="nv">$migration_file</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">-u</span> <span class="s2">"http://</span><span class="nv">$source_env</span><span class="s2">:8088/"</span> <span class="se">\</span>
<span class="nt">--collection</span> <span class="s2">"</span><span class="nv">$collection</span><span class="s2">"</span> &
<span class="nv">export_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="c"># Let it run for specified time to capture data</span>
<span class="nb">echo</span> <span class="s2">"Capturing embeddings for migration..."</span>
<span class="nb">echo</span> <span class="s2">"Process PID: </span><span class="nv">$export_pid</span><span class="s2">"</span>
<span class="c"># In practice, you'd run this for the duration needed</span>
<span class="c"># sleep 1800 # 30 minutes</span>
<span class="c"># kill $export_pid</span>
<span class="nb">echo</span> <span class="s2">"Migration data will be saved to: </span><span class="nv">$migration_file</span><span class="s2">"</span>
<span class="o">}</span>
<span class="c"># Prepare migration from dev to production</span>
prepare_migration_data <span class="s2">"dev-server"</span> <span class="s2">"processed-docs"</span>
</code></pre></div></div>
<h3 id="continuous-export">
<a href="#continuous-export" class="anchor-heading" aria-labelledby="continuous-export"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Continuous Export
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Continuous embeddings export with rotation</span>
continuous_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">output_dir</span><span class="o">=</span><span class="s2">"continuous-exports"</span>
<span class="nb">local </span><span class="nv">rotation_hours</span><span class="o">=</span>24
<span class="nb">local </span><span class="nv">file_prefix</span><span class="o">=</span><span class="s2">"embeddings"</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> <span class="s2">"</span><span class="nv">$output_dir</span><span class="s2">"</span>
<span class="k">while </span><span class="nb">true</span><span class="p">;</span> <span class="k">do
</span><span class="nv">timestamp</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d_%H%M%S<span class="si">)</span>
<span class="nv">output_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$output_dir</span><span class="s2">/</span><span class="k">${</span><span class="nv">file_prefix</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="nv">timestamp</span><span class="k">}</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Starting export to: </span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="c"># Start export for specified duration</span>
<span class="nb">timeout</span> <span class="k">${</span><span class="nv">rotation_hours</span><span class="k">}</span>h tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="c"># Compress completed file</span>
<span class="nb">gzip</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Export completed and compressed: </span><span class="k">${</span><span class="nv">output_file</span><span class="k">}</span><span class="s2">.gz"</span>
<span class="c"># Optional: clean up old files</span>
find <span class="s2">"</span><span class="nv">$output_dir</span><span class="s2">"</span> <span class="nt">-name</span> <span class="s2">"*.msgpack.gz"</span> <span class="nt">-mtime</span> +30 <span class="nt">-delete</span>
<span class="c"># Brief pause before next rotation</span>
<span class="nb">sleep </span>60
<span class="k">done</span>
<span class="o">}</span>
<span class="c"># Start continuous export (run in background)</span>
continuous_export &
</code></pre></div></div>
<h3 id="analysis-and-research">
<a href="#analysis-and-research" class="anchor-heading" aria-labelledby="analysis-and-research"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Analysis and Research
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Export embeddings for research analysis</span>
export_for_research<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">research_topic</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">output_file</span><span class="o">=</span><span class="s2">"research-</span><span class="k">${</span><span class="nv">research_topic</span><span class="k">}</span><span class="s2">-</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d<span class="si">)</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Exporting embeddings for research: </span><span class="nv">$research_topic</span><span class="s2">"</span>
<span class="c"># Start export with filtering</span>
tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--collection</span> <span class="s2">"</span><span class="nv">$research_topic</span><span class="s2">"</span> &
<span class="nv">export_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="nb">echo</span> <span class="s2">"Research export started (PID: </span><span class="nv">$export_pid</span><span class="s2">)"</span>
<span class="nb">echo</span> <span class="s2">"Output: </span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="c"># Create analysis script</span>
<span class="nb">cat</span> <span class="o">></span> <span class="s2">"analyze-</span><span class="k">${</span><span class="nv">research_topic</span><span class="k">}</span><span class="s2">.sh"</span> <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
#!/bin/bash
# Analysis script for </span><span class="nv">$research_topic</span><span class="sh"> embeddings
echo "Analyzing </span><span class="nv">$research_topic</span><span class="sh"> embeddings..."
# Basic statistics
echo "=== Basic Statistics ==="
tg-dump-msgpack -i "</span><span class="nv">$output_file</span><span class="sh">" --summary
# Detailed analysis
echo "=== Detailed Analysis ==="
tg-dump-msgpack -i "</span><span class="nv">$output_file</span><span class="sh">" | head -10
echo "Analysis complete for </span><span class="nv">$research_topic</span><span class="sh">"
</span><span class="no">EOF
</span> <span class="nb">chmod</span> +x <span class="s2">"analyze-</span><span class="k">${</span><span class="nv">research_topic</span><span class="k">}</span><span class="s2">.sh"</span>
<span class="nb">echo</span> <span class="s2">"Analysis script created: analyze-</span><span class="k">${</span><span class="nv">research_topic</span><span class="k">}</span><span class="s2">.sh"</span>
<span class="o">}</span>
<span class="c"># Export for different research topics</span>
export_for_research <span class="s2">"cybersecurity"</span>
export_for_research <span class="s2">"climate-change"</span>
</code></pre></div></div>
<h2 id="advanced-usage">
<a href="#advanced-usage" class="anchor-heading" aria-labelledby="advanced-usage"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Advanced Usage
</h2>
<h3 id="selective-export">
<a href="#selective-export" class="anchor-heading" aria-labelledby="selective-export"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Selective Export
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Export embeddings with multiple filters</span>
selective_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local users</span><span class="o">=(</span><span class="s2">"user1"</span> <span class="s2">"user2"</span> <span class="s2">"user3"</span><span class="o">)</span>
<span class="nb">local </span><span class="nv">collections</span><span class="o">=(</span><span class="s2">"docs1"</span> <span class="s2">"docs2"</span><span class="o">)</span>
<span class="k">for </span>user <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="nv">users</span><span class="p">[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
for </span>collection <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="nv">collections</span><span class="p">[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
</span><span class="nv">output_file</span><span class="o">=</span><span class="s2">"embeddings-</span><span class="k">${</span><span class="nv">user</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="nv">collection</span><span class="k">}</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Exporting for user: </span><span class="nv">$user</span><span class="s2">, collection: </span><span class="nv">$collection</span><span class="s2">"</span>
tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--user</span> <span class="s2">"</span><span class="nv">$user</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--collection</span> <span class="s2">"</span><span class="nv">$collection</span><span class="s2">"</span> &
<span class="c"># Store PID for later management</span>
<span class="nb">echo</span> <span class="nv">$!</span> <span class="o">></span> <span class="s2">"</span><span class="k">${</span><span class="nv">output_file</span><span class="k">}</span><span class="s2">.pid"</span>
<span class="k">done
done
</span><span class="nb">echo</span> <span class="s2">"All selective exports started"</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="monitoring-and-statistics">
<a href="#monitoring-and-statistics" class="anchor-heading" aria-labelledby="monitoring-and-statistics"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Monitoring and Statistics
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Monitor export progress with statistics</span>
monitor_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">output_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">pid_file</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">output_file</span><span class="k">}</span><span class="s2">.pid"</span>
<span class="k">if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$pid_file</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"PID file not found: </span><span class="nv">$pid_file</span><span class="s2">"</span>
<span class="k">return </span>1
<span class="k">fi
</span><span class="nb">local </span><span class="nv">export_pid</span><span class="o">=</span><span class="si">$(</span><span class="nb">cat</span> <span class="s2">"</span><span class="nv">$pid_file</span><span class="s2">"</span><span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"Monitoring export (PID: </span><span class="nv">$export_pid</span><span class="s2">)..."</span>
<span class="nb">echo</span> <span class="s2">"Output file: </span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="k">while </span><span class="nb">kill</span> <span class="nt">-0</span> <span class="s2">"</span><span class="nv">$export_pid</span><span class="s2">"</span> 2>/dev/null<span class="p">;</span> <span class="k">do
if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nv">file_size</span><span class="o">=</span><span class="si">$(</span><span class="nb">stat</span> <span class="nt">-c</span>%s <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> 2>/dev/null <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"0"</span><span class="si">)</span>
<span class="nv">human_size</span><span class="o">=</span><span class="si">$(</span><span class="nb">numfmt</span> <span class="nt">--to</span><span class="o">=</span>iec-i <span class="nt">--suffix</span><span class="o">=</span>B <span class="s2">"</span><span class="nv">$file_size</span><span class="s2">"</span><span class="si">)</span>
<span class="c"># Try to count embeddings</span>
<span class="nv">embedding_count</span><span class="o">=</span><span class="si">$(</span>tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> 2>/dev/null | <span class="nb">grep</span> <span class="nt">-c</span> <span class="s1">'^\["de"'</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"0"</span><span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"File size: </span><span class="nv">$human_size</span><span class="s2">, Embeddings: </span><span class="nv">$embedding_count</span><span class="s2">"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"Output file not yet created..."</span>
<span class="k">fi
</span><span class="nb">sleep </span>30
<span class="k">done
</span><span class="nb">echo</span> <span class="s2">"Export process completed"</span>
<span class="nb">rm</span> <span class="s2">"</span><span class="nv">$pid_file</span><span class="s2">"</span>
<span class="o">}</span>
<span class="c"># Start export and monitor</span>
tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"monitored-export.msgpack"</span> &
<span class="nb">echo</span> <span class="nv">$!</span> <span class="o">></span> <span class="s2">"monitored-export.msgpack.pid"</span>
monitor_export <span class="s2">"monitored-export.msgpack"</span>
</code></pre></div></div>
<h3 id="export-validation">
<a href="#export-validation" class="anchor-heading" aria-labelledby="export-validation"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Export Validation
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Validate exported embeddings</span>
validate_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">export_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Validating export file: </span><span class="nv">$export_file</span><span class="s2">"</span>
<span class="c"># Check file exists and has content</span>
<span class="k">if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-s</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"✗ Export file is empty or missing"</span>
<span class="k">return </span>1
<span class="k">fi</span>
<span class="c"># Check MessagePack format</span>
<span class="k">if </span>tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> <span class="nt">--summary</span> <span class="o">></span> /dev/null 2>&1<span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"✓ Valid MessagePack format"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"✗ Invalid MessagePack format"</span>
<span class="k">return </span>1
<span class="k">fi</span>
<span class="c"># Check for document embeddings</span>
<span class="nv">embedding_count</span><span class="o">=</span><span class="si">$(</span>tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> | <span class="nb">grep</span> <span class="nt">-c</span> <span class="s1">'^\["de"'</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"0"</span><span class="si">)</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$embedding_count</span><span class="s2">"</span> <span class="nt">-gt</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"✓ Contains </span><span class="nv">$embedding_count</span><span class="s2"> document embeddings"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"✗ No document embeddings found"</span>
<span class="k">return </span>1
<span class="k">fi</span>
<span class="c"># Get vector dimension information</span>
<span class="nv">summary</span><span class="o">=</span><span class="si">$(</span>tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> <span class="nt">--summary</span><span class="si">)</span>
<span class="k">if </span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$summary</span><span class="s2">"</span> | <span class="nb">grep</span> <span class="nt">-q</span> <span class="s2">"Vector dimension:"</span><span class="p">;</span> <span class="k">then
</span><span class="nv">dimension</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$summary</span><span class="s2">"</span> | <span class="nb">grep</span> <span class="s2">"Vector dimension:"</span> | <span class="nb">awk</span> <span class="s1">'{print $3}'</span><span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"✓ Vector dimension: </span><span class="nv">$dimension</span><span class="s2">"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"⚠ Could not determine vector dimension"</span>
<span class="k">fi
</span><span class="nb">echo</span> <span class="s2">"Validation completed successfully"</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="export-scheduling">
<a href="#export-scheduling" class="anchor-heading" aria-labelledby="export-scheduling"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Export Scheduling
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Scheduled export with cron-like functionality</span>
schedule_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">schedule</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="c"># e.g., "daily", "hourly", "weekly"</span>
<span class="nb">local </span><span class="nv">output_prefix</span><span class="o">=</span><span class="s2">"</span><span class="nv">$2</span><span class="s2">"</span>
<span class="k">case</span> <span class="s2">"</span><span class="nv">$schedule</span><span class="s2">"</span> <span class="k">in</span>
<span class="s2">"hourly"</span><span class="p">)</span>
<span class="nv">interval</span><span class="o">=</span>3600
<span class="p">;;</span>
<span class="s2">"daily"</span><span class="p">)</span>
<span class="nv">interval</span><span class="o">=</span>86400
<span class="p">;;</span>
<span class="s2">"weekly"</span><span class="p">)</span>
<span class="nv">interval</span><span class="o">=</span>604800
<span class="p">;;</span>
<span class="k">*</span><span class="p">)</span>
<span class="nb">echo</span> <span class="s2">"Invalid schedule: </span><span class="nv">$schedule</span><span class="s2">"</span>
<span class="k">return </span>1
<span class="p">;;</span>
<span class="k">esac</span>
<span class="nb">echo</span> <span class="s2">"Starting </span><span class="nv">$schedule</span><span class="s2"> exports with prefix: </span><span class="nv">$output_prefix</span><span class="s2">"</span>
<span class="k">while </span><span class="nb">true</span><span class="p">;</span> <span class="k">do
</span><span class="nv">timestamp</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d_%H%M%S<span class="si">)</span>
<span class="nv">output_file</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">output_prefix</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="nv">timestamp</span><span class="k">}</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Starting scheduled export: </span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="c"># Run export for the scheduled interval</span>
<span class="nb">timeout</span> <span class="k">${</span><span class="nv">interval</span><span class="k">}</span>s tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="c"># Validate and compress</span>
<span class="k">if </span>validate_export <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span><span class="p">;</span> <span class="k">then
</span><span class="nb">gzip</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"✓ Export completed and compressed: </span><span class="k">${</span><span class="nv">output_file</span><span class="k">}</span><span class="s2">.gz"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"✗ Export validation failed: </span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="nb">mv</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> <span class="s2">"</span><span class="k">${</span><span class="nv">output_file</span><span class="k">}</span><span class="s2">.failed"</span>
<span class="k">fi</span>
<span class="c"># Brief pause before next cycle</span>
<span class="nb">sleep </span>60
<span class="k">done</span>
<span class="o">}</span>
<span class="c"># Start daily scheduled exports</span>
schedule_export <span class="s2">"daily"</span> <span class="s2">"daily-embeddings"</span> &
</code></pre></div></div>
<h2 id="performance-considerations">
<a href="#performance-considerations" class="anchor-heading" aria-labelledby="performance-considerations"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Performance Considerations
</h2>
<h3 id="memory-management">
<a href="#memory-management" class="anchor-heading" aria-labelledby="memory-management"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Memory Management
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Monitor memory usage during export</span>
monitor_memory_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">output_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="c"># Start export</span>
tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> &
<span class="nv">export_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="nb">echo</span> <span class="s2">"Monitoring memory usage for export (PID: </span><span class="nv">$export_pid</span><span class="s2">)..."</span>
<span class="k">while </span><span class="nb">kill</span> <span class="nt">-0</span> <span class="s2">"</span><span class="nv">$export_pid</span><span class="s2">"</span> 2>/dev/null<span class="p">;</span> <span class="k">do
</span><span class="nv">memory_usage</span><span class="o">=</span><span class="si">$(</span>ps <span class="nt">-p</span> <span class="s2">"</span><span class="nv">$export_pid</span><span class="s2">"</span> <span class="nt">-o</span> <span class="nv">rss</span><span class="o">=</span> 2>/dev/null | <span class="nb">awk</span> <span class="s1">'{print $1/1024}'</span><span class="si">)</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-n</span> <span class="s2">"</span><span class="nv">$memory_usage</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Memory usage: </span><span class="k">${</span><span class="nv">memory_usage</span><span class="k">}</span><span class="s2">MB"</span>
<span class="k">fi
</span><span class="nb">sleep </span>10
<span class="k">done
</span><span class="nb">echo</span> <span class="s2">"Export completed"</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="network-optimization">
<a href="#network-optimization" class="anchor-heading" aria-labelledby="network-optimization"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Network Optimization
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Optimize for network conditions</span>
network_optimized_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">output_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">api_url</span><span class="o">=</span><span class="s2">"</span><span class="nv">$2</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Starting network-optimized export..."</span>
<span class="c"># Use compression and buffering</span>
tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">-u</span> <span class="s2">"</span><span class="nv">$api_url</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--format</span> msgpack & <span class="c"># MessagePack is more compact than JSON</span>
<span class="nv">export_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="c"># Monitor network usage</span>
<span class="nb">echo</span> <span class="s2">"Monitoring export (PID: </span><span class="nv">$export_pid</span><span class="s2">)..."</span>
<span class="k">while </span><span class="nb">kill</span> <span class="nt">-0</span> <span class="s2">"</span><span class="nv">$export_pid</span><span class="s2">"</span> 2>/dev/null<span class="p">;</span> <span class="k">do</span>
<span class="c"># Monitor network connections</span>
<span class="nv">connections</span><span class="o">=</span><span class="si">$(</span>netstat <span class="nt">-an</span> | <span class="nb">grep</span> <span class="s2">":8088"</span> | <span class="nb">wc</span> <span class="nt">-l</span><span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"Active connections: </span><span class="nv">$connections</span><span class="s2">"</span>
<span class="nb">sleep </span>30
<span class="k">done</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="error-handling">
<a href="#error-handling" class="anchor-heading" aria-labelledby="error-handling"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Error Handling
</h2>
<h3 id="connection-issues">
<a href="#connection-issues" class="anchor-heading" aria-labelledby="connection-issues"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Connection Issues
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Exception: WebSocket connection failed
</code></pre></div></div>
<p><strong>Solution</strong>: Check API URL and ensure TrustGraph WebSocket service is running.</p>
<h3 id="disk-space-issues">
<a href="#disk-space-issues" class="anchor-heading" aria-labelledby="disk-space-issues"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Disk Space Issues
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Exception: No space left on device
</code></pre></div></div>
<p><strong>Solution</strong>: Free up disk space or use a different output location.</p>
<h3 id="permission-errors">
<a href="#permission-errors" class="anchor-heading" aria-labelledby="permission-errors"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Permission Errors
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Exception: Permission denied
</code></pre></div></div>
<p><strong>Solution</strong>: Check write permissions for the output file location.</p>
<h3 id="memory-issues">
<a href="#memory-issues" class="anchor-heading" aria-labelledby="memory-issues"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Memory Issues
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MemoryError: Unable to allocate memory
</code></pre></div></div>
<p><strong>Solution</strong>: Monitor memory usage and consider using smaller export windows.</p>
<h2 id="integration-with-other-commands">
<a href="#integration-with-other-commands" class="anchor-heading" aria-labelledby="integration-with-other-commands"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Integration with Other Commands
</h2>
<h3 id="complete-backup-workflow">
<a href="#complete-backup-workflow" class="anchor-heading" aria-labelledby="complete-backup-workflow"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Complete Backup Workflow
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Complete backup and restore workflow</span>
backup_restore_workflow<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">backup_file</span><span class="o">=</span><span class="s2">"embeddings-backup.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"=== Backup Phase ==="</span>
<span class="c"># Create backup</span>
tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"</span><span class="nv">$backup_file</span><span class="s2">"</span> &
<span class="nv">backup_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="c"># Let it run for a while</span>
<span class="nb">sleep </span>300 <span class="c"># 5 minutes</span>
<span class="nb">kill</span> <span class="nv">$backup_pid</span>
<span class="nb">echo</span> <span class="s2">"Backup created: </span><span class="nv">$backup_file</span><span class="s2">"</span>
<span class="c"># Validate backup</span>
validate_export <span class="s2">"</span><span class="nv">$backup_file</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"=== Restore Phase ==="</span>
<span class="c"># Restore from backup (to different collection)</span>
tg-load-doc-embeds <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$backup_file</span><span class="s2">"</span> <span class="nt">--collection</span> <span class="s2">"restored"</span>
<span class="nb">echo</span> <span class="s2">"Backup and restore workflow completed"</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="analysis-pipeline">
<a href="#analysis-pipeline" class="anchor-heading" aria-labelledby="analysis-pipeline"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Analysis Pipeline
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Export and analyze embeddings</span>
export_analyze_pipeline<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">topic</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">export_file</span><span class="o">=</span><span class="s2">"analysis-</span><span class="k">${</span><span class="nv">topic</span><span class="k">}</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Starting export and analysis pipeline for: </span><span class="nv">$topic</span><span class="s2">"</span>
<span class="c"># Export embeddings</span>
tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--collection</span> <span class="s2">"</span><span class="nv">$topic</span><span class="s2">"</span> &
<span class="nv">export_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="c"># Run for analysis duration</span>
<span class="nb">sleep </span>600 <span class="c"># 10 minutes</span>
<span class="nb">kill</span> <span class="nv">$export_pid</span>
<span class="c"># Analyze exported data</span>
<span class="nb">echo</span> <span class="s2">"Analyzing exported embeddings..."</span>
tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> <span class="nt">--summary</span>
<span class="c"># Count embeddings by user</span>
<span class="nb">echo</span> <span class="s2">"Embeddings by user:"</span>
tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> | <span class="se">\</span>
jq <span class="nt">-r</span> <span class="s1">'.[1].m.u'</span> | <span class="se">\</span>
<span class="nb">sort</span> | <span class="nb">uniq</span> <span class="nt">-c</span>
<span class="nb">echo</span> <span class="s2">"Analysis pipeline completed"</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="environment-variables">
<a href="#environment-variables" class="anchor-heading" aria-labelledby="environment-variables"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Environment Variables
</h2>
<ul>
<li><code class="language-plaintext highlighter-rouge">TRUSTGRAPH_API</code>: Default API URL</li>
</ul>
<h2 id="related-commands">
<a href="#related-commands" class="anchor-heading" aria-labelledby="related-commands"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Related Commands
</h2>
<ul>
<li><a href="tg-load-doc-embeds"><code class="language-plaintext highlighter-rouge">tg-load-doc-embeds</code></a> - Load document embeddings from files</li>
<li><a href="tg-dump-msgpack"><code class="language-plaintext highlighter-rouge">tg-dump-msgpack</code></a> - Analyze MessagePack files</li>
<li><a href="tg-show-flows"><code class="language-plaintext highlighter-rouge">tg-show-flows</code></a> - List available flows for monitoring</li>
</ul>
<h2 id="api-integration">
<a href="#api-integration" class="anchor-heading" aria-labelledby="api-integration"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> API Integration
</h2>
<p>This command uses TrustGraph’s WebSocket API for document embeddings export, specifically the <code class="language-plaintext highlighter-rouge">/api/v1/flow/{flow-id}/export/document-embeddings</code> endpoint.</p>
<h2 id="best-practices">
<a href="#best-practices" class="anchor-heading" aria-labelledby="best-practices"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Best Practices
</h2>
<ol>
<li><strong>Start Early</strong>: Begin export before processing starts to capture all data</li>
<li><strong>Monitoring</strong>: Monitor export progress and file sizes</li>
<li><strong>Validation</strong>: Always validate exported files</li>
<li><strong>Compression</strong>: Use compression for long-term storage</li>
<li><strong>Rotation</strong>: Implement file rotation for continuous exports</li>
<li><strong>Backup</strong>: Keep multiple backup copies in different locations</li>
<li><strong>Documentation</strong>: Document export schedules and procedures</li>
</ol>
<h2 id="troubleshooting">
<a href="#troubleshooting" class="anchor-heading" aria-labelledby="troubleshooting"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Troubleshooting
</h2>
<h3 id="no-data-captured">
<a href="#no-data-captured" class="anchor-heading" aria-labelledby="no-data-captured"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> No Data Captured
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Check if processing is generating embeddings</span>
tg-show-flows | <span class="nb">grep </span>processing
<span class="c"># Verify WebSocket connection</span>
netstat <span class="nt">-an</span> | <span class="nb">grep</span> :8088
</code></pre></div></div>
<h3 id="large-file-issues">
<a href="#large-file-issues" class="anchor-heading" aria-labelledby="large-file-issues"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Large File Issues
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Monitor file growth</span>
watch <span class="nt">-n</span> 5 <span class="s1">'ls -lh *.msgpack'</span>
<span class="c"># Check available disk space</span>
<span class="nb">df</span> <span class="nt">-h</span>
</code></pre></div></div>
<h3 id="process-management">
<a href="#process-management" class="anchor-heading" aria-labelledby="process-management"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Process Management
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># List running export processes</span>
ps aux | <span class="nb">grep </span>tg-save-doc-embeds
<span class="c"># Kill stuck processes</span>
pkill <span class="nt">-f</span> tg-save-doc-embeds
</code></pre></div></div>
</main>
<hr>
<footer>
<p class="text-small text-grey-dk-100 mb-0">Copyright © 2024-2025 TrustGraph.ai. Distributed under an <a href="https://github.com/trustgraph-ai/trustgraph/blob/master/LICENSE">Apache 2 license.</a></p>
<div class="d-flex mt-2">
<p class="text-small text-grey-dk-000 mb-0">
<a href="https://github.com/trustgraph-ai/trustgraph-ai.github.io/tree/main/reference/cli/tg-save-doc-embeds.md" id="edit-this-page">Edit this page on GitHub.</a>
</p>
</div>
</footer>
</div>
</div>
<div class="search-overlay"></div>
</div>
<script src="https://cdn.jsdelivr.net/npm/mermaid@9.1.3/dist/mermaid.min.js"></script>
<script>
var config = {}
;
mermaid.initialize(config);
window.mermaid.init(undefined, document.querySelectorAll('.language-mermaid'));
</script>
</body>
</html>
"
# Better: Include context and constraints
tg-set-prompt \
--id "better-summary" \
--prompt "Task: Summarize the following
Length: words maximum
Focus:
Audience:
Document:
<!DOCTYPE html>
<html lang="en-US">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=Edge">
<link rel="stylesheet" href="/assets/css/just-the-docs-default.css">
<link rel="stylesheet" href="/assets/css/just-the-docs-head-nav.css" id="jtd-head-nav-stylesheet">
<style id="jtd-nav-activation">
.site-nav > ul.nav-list:first-child > li > a,
.site-nav > ul.nav-list:first-child > li > ul > li > a,
.site-nav > ul.nav-list:first-child > li > ul > li > ul > li > a,
.site-nav > ul.nav-list:first-child > li > ul > li > ul > li > ul > li:not(:nth-child(24)) > a,
.site-nav > ul.nav-list:first-child > li > ul > li > ul > li > ul > li > ul > li a {
background-image: none;
}
.site-nav > ul.nav-list:not(:first-child) a,
.site-nav li.external a {
background-image: none;
}
.site-nav > ul.nav-list:first-child > li:nth-child(1) > ul > li:nth-child(5) > ul > li:nth-child(1) > ul > li:nth-child(24) > a {
font-weight: 600;
text-decoration: none;
}.site-nav > ul.nav-list:first-child > li:nth-child(1) > button svg,
.site-nav > ul.nav-list:first-child > li:nth-child(1) > ul > li:nth-child(5) > button svg,
.site-nav > ul.nav-list:first-child > li:nth-child(1) > ul > li:nth-child(5) > ul > li:nth-child(1) > button svg,
.site-nav > ul.nav-list:first-child > li:nth-child(1) > ul > li:nth-child(5) > ul > li:nth-child(1) > ul > li:nth-child(24) > button svg {
transform: rotate(-90deg);
}.site-nav > ul.nav-list:first-child > li.nav-list-item:nth-child(1) > ul.nav-list,
.site-nav > ul.nav-list:first-child > li.nav-list-item:nth-child(1) > ul.nav-list > li.nav-list-item:nth-child(5) > ul.nav-list,
.site-nav > ul.nav-list:first-child > li.nav-list-item:nth-child(1) > ul.nav-list > li.nav-list-item:nth-child(5) > ul.nav-list > li.nav-list-item:nth-child(1) > ul.nav-list,
.site-nav > ul.nav-list:first-child > li.nav-list-item:nth-child(1) > ul.nav-list > li.nav-list-item:nth-child(5) > ul.nav-list > li.nav-list-item:nth-child(1) > ul.nav-list > li.nav-list-item:nth-child(24) > ul.nav-list {
display: block;
}
</style>
<script src="/assets/js/vendor/lunr.min.js"></script>
<script src="/assets/js/just-the-docs.js"></script>
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- Begin Jekyll SEO tag v2.8.0 -->
<title>tg-save-doc-embeds | TrustGraph</title>
<meta name="generator" content="Jekyll v4.4.1" />
<meta property="og:title" content="tg-save-doc-embeds" />
<meta property="og:locale" content="en_US" />
<meta name="description" content="Information on the TrustGraph platform" />
<meta property="og:description" content="Information on the TrustGraph platform" />
<link rel="canonical" href="https://just-the-docs.github.io/reference/cli/tg-save-doc-embeds.html" />
<meta property="og:url" content="https://just-the-docs.github.io/reference/cli/tg-save-doc-embeds.html" />
<meta property="og:site_name" content="TrustGraph" />
<meta property="og:type" content="website" />
<meta name="twitter:card" content="summary" />
<meta property="twitter:title" content="tg-save-doc-embeds" />
<script type="application/ld+json">
{"@context":"https://schema.org","@type":"WebPage","description":"Information on the TrustGraph platform","headline":"tg-save-doc-embeds","url":"https://just-the-docs.github.io/reference/cli/tg-save-doc-embeds.html"}</script>
<!-- End Jekyll SEO tag -->
</head>
<body>
<a class="skip-to-main" href="#main-content">Skip to main content</a>
<svg xmlns="http://www.w3.org/2000/svg" class="d-none">
<symbol id="svg-link" viewBox="0 0 24 24">
<title>Link</title>
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-link">
<path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"></path><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"></path>
</svg>
</symbol>
<symbol id="svg-menu" viewBox="0 0 24 24">
<title>Menu</title>
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-menu">
<line x1="3" y1="12" x2="21" y2="12"></line><line x1="3" y1="6" x2="21" y2="6"></line><line x1="3" y1="18" x2="21" y2="18"></line>
</svg>
</symbol>
<symbol id="svg-arrow-right" viewBox="0 0 24 24">
<title>Expand</title>
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-chevron-right">
<polyline points="9 18 15 12 9 6"></polyline>
</svg>
</symbol>
<!-- Feather. MIT License: https://github.com/feathericons/feather/blob/master/LICENSE -->
<symbol id="svg-external-link" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-external-link">
<title id="svg-external-link-title">(external link)</title>
<path d="M18 13v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"></path><polyline points="15 3 21 3 21 9"></polyline><line x1="10" y1="14" x2="21" y2="3"></line>
</symbol>
<symbol id="svg-doc" viewBox="0 0 24 24">
<title>Document</title>
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-file">
<path d="M13 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V9z"></path><polyline points="13 2 13 9 20 9"></polyline>
</svg>
</symbol>
<symbol id="svg-search" viewBox="0 0 24 24">
<title>Search</title>
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-search">
<circle cx="11" cy="11" r="8"></circle><line x1="21" y1="21" x2="16.65" y2="16.65"></line>
</svg>
</symbol>
<!-- Bootstrap Icons. MIT License: https://github.com/twbs/icons/blob/main/LICENSE.md -->
<symbol id="svg-copy" viewBox="0 0 16 16">
<title>Copy</title>
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentColor" class="bi bi-clipboard" viewBox="0 0 16 16">
<path d="M4 1.5H3a2 2 0 0 0-2 2V14a2 2 0 0 0 2 2h10a2 2 0 0 0 2-2V3.5a2 2 0 0 0-2-2h-1v1h1a1 1 0 0 1 1 1V14a1 1 0 0 1-1 1H3a1 1 0 0 1-1-1V3.5a1 1 0 0 1 1-1h1v-1z"/>
<path d="M9.5 1a.5.5 0 0 1 .5.5v1a.5.5 0 0 1-.5.5h-3a.5.5 0 0 1-.5-.5v-1a.5.5 0 0 1 .5-.5h3zm-3-1A1.5 1.5 0 0 0 5 1.5v1A1.5 1.5 0 0 0 6.5 4h3A1.5 1.5 0 0 0 11 2.5v-1A1.5 1.5 0 0 0 9.5 0h-3z"/>
</svg>
</symbol>
<symbol id="svg-copied" viewBox="0 0 16 16">
<title>Copied</title>
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentColor" class="bi bi-clipboard-check-fill" viewBox="0 0 16 16">
<path d="M6.5 0A1.5 1.5 0 0 0 5 1.5v1A1.5 1.5 0 0 0 6.5 4h3A1.5 1.5 0 0 0 11 2.5v-1A1.5 1.5 0 0 0 9.5 0h-3Zm3 1a.5.5 0 0 1 .5.5v1a.5.5 0 0 1-.5.5h-3a.5.5 0 0 1-.5-.5v-1a.5.5 0 0 1 .5-.5h3Z"/>
<path d="M4 1.5H3a2 2 0 0 0-2 2V14a2 2 0 0 0 2 2h10a2 2 0 0 0 2-2V3.5a2 2 0 0 0-2-2h-1v1A2.5 2.5 0 0 1 9.5 5h-3A2.5 2.5 0 0 1 4 2.5v-1Zm6.854 7.354-3 3a.5.5 0 0 1-.708 0l-1.5-1.5a.5.5 0 0 1 .708-.708L7.5 10.793l2.646-2.647a.5.5 0 0 1 .708.708Z"/>
</svg>
</symbol>
</svg>
<div class="side-bar">
<div class="site-header" role="banner">
<a href="/" class="site-title lh-tight">
TrustGraph
</a>
<button id="menu-button" class="site-button btn-reset" aria-label="Toggle menu" aria-pressed="false">
<svg viewBox="0 0 24 24" class="icon" aria-hidden="true"><use xlink:href="#svg-menu"></use></svg>
</button>
</div>
<nav aria-label="Main" id="site-nav" class="site-nav">
<ul class="nav-list"><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in TrustGraph Documentation category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/" class="nav-list-link">TrustGraph Documentation</a><ul class="nav-list"><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Getting Started category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/getting-started/" class="nav-list-link">Getting Started</a><ul class="nav-list"><li class="nav-list-item"><a href="/getting-started/concepts.html" class="nav-list-link">Core Concepts</a></li><li class="nav-list-item"><a href="/getting-started/installation.html" class="nav-list-link">Installation</a></li><li class="nav-list-item"><a href="/getting-started/first-steps.html" class="nav-list-link">First Steps</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Overview category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/overview/" class="nav-list-link">Overview</a><ul class="nav-list"><li class="nav-list-item"><a href="/overview/philosophy.html" class="nav-list-link">Philosophy</a></li><li class="nav-list-item"><a href="/overview/architecture.html" class="nav-list-link">Architecture</a></li><li class="nav-list-item"><a href="/overview/features.html" class="nav-list-link">Features</a></li><li class="nav-list-item"><a href="/overview/use-cases.html" class="nav-list-link">Use Cases</a></li><li class="nav-list-item"><a href="/overview/feature-maturity.html" class="nav-list-link">Feature Maturity</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Deployment category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/deployment/" class="nav-list-link">Deployment</a><ul class="nav-list"><li class="nav-list-item"><a href="/deployment/docker-compose.html" class="nav-list-link">Docker Compose / Podman Compose</a></li><li class="nav-list-item"><a href="/deployment/minikube.html" class="nav-list-link">Minikube</a></li><li class="nav-list-item"><a href="/deployment/intel.html" class="nav-list-link">Intel GPU / Tiber Cloud</a></li><li class="nav-list-item"><a href="/deployment/scaleway.html" class="nav-list-link">Scaleway</a></li><li class="nav-list-item"><a href="/deployment/azure.html" class="nav-list-link">Azure AKS</a></li><li class="nav-list-item"><a href="/deployment/gcp.html" class="nav-list-link">Google Cloud Platform</a></li><li class="nav-list-item"><a href="/deployment/aws-ec2.html" class="nav-list-link">AWS EC2 Single Instance</a></li><li class="nav-list-item"><a href="/deployment/aws-rke.html" class="nav-list-link">Amazon Web Services (RKE)</a></li><li class="nav-list-item"><a href="/deployment/security-considerations.html" class="nav-list-link">Security Considerations</a></li><li class="nav-list-item"><a href="/deployment/production-considerations.html" class="nav-list-link">Production Considerations</a></li><li class="nav-list-item"><a href="/deployment/troubleshooting.html" class="nav-list-link">Troubleshooting</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in How-to Guides category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/guides/" class="nav-list-link">How-to Guides</a><ul class="nav-list"><li class="nav-list-item"><a href="/guides/data-integration/" class="nav-list-link">Data Integration</a></li><li class="nav-list-item"><a href="/guides/mcp-integration/" class="nav-list-link">MCP Integration</a></li><li class="nav-list-item"><a href="/guides/migration/" class="nav-list-link">Migration</a></li><li class="nav-list-item"><a href="/guides/monitoring/" class="nav-list-link">Monitoring</a></li><li class="nav-list-item"><a href="/guides/querying/" class="nav-list-link">Querying</a></li><li class="nav-list-item"><a href="/guides/security/" class="nav-list-link">Security</a></li><li class="nav-list-item"><a href="/guides/visualization/" class="nav-list-link">Visualization</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Reference category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/reference/" class="nav-list-link">Reference</a><ul class="nav-list"><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in CLI category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/reference/cli/" class="nav-list-link">CLI</a><ul class="nav-list"><li class="nav-list-item"><a href="/reference/cli/tg-add-library-document.html" class="nav-list-link">tg-add-library-document</a></li><li class="nav-list-item"><a href="/reference/cli/tg-delete-flow-class.html" class="nav-list-link">tg-delete-flow-class</a></li><li class="nav-list-item"><a href="/reference/cli/tg-delete-kg-core.html" class="nav-list-link">tg-delete-kg-core</a></li><li class="nav-list-item"><a href="/reference/cli/tg-dump-msgpack.html" class="nav-list-link">tg-dump-msgpack</a></li><li class="nav-list-item"><a href="/reference/cli/tg-get-flow-class.html" class="nav-list-link">tg-get-flow-class</a></li><li class="nav-list-item"><a href="/reference/cli/tg-get-kg-core.html" class="nav-list-link">tg-get-kg-core</a></li><li class="nav-list-item"><a href="/reference/cli/tg-graph-to-turtle.html" class="nav-list-link">tg-graph-to-turtle</a></li><li class="nav-list-item"><a href="/reference/cli/tg-init-pulsar-manager.html" class="nav-list-link">tg-init-pulsar-manager</a></li><li class="nav-list-item"><a href="/reference/cli/tg-init-trustgraph.html" class="nav-list-link">tg-init-trustgraph</a></li><li class="nav-list-item"><a href="/reference/cli/tg-invoke-agent.html" class="nav-list-link">tg-invoke-agent</a></li><li class="nav-list-item"><a href="/reference/cli/tg-invoke-document-rag.html" class="nav-list-link">tg-invoke-document-rag</a></li><li class="nav-list-item"><a href="/reference/cli/tg-invoke-graph-rag.html" class="nav-list-link">tg-invoke-graph-rag</a></li><li class="nav-list-item"><a href="/reference/cli/tg-invoke-llm.html" class="nav-list-link">tg-invoke-llm</a></li><li class="nav-list-item"><a href="/reference/cli/tg-invoke-prompt.html" class="nav-list-link">tg-invoke-prompt</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-doc-embeds.html" class="nav-list-link">tg-load-doc-embeds</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-kg-core.html" class="nav-list-link">tg-load-kg-core</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-pdf.html" class="nav-list-link">tg-load-pdf</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-sample-documents.html" class="nav-list-link">tg-load-sample-documents</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-text.html" class="nav-list-link">tg-load-text</a></li><li class="nav-list-item"><a href="/reference/cli/tg-load-turtle.html" class="nav-list-link">tg-load-turtle</a></li><li class="nav-list-item"><a href="/reference/cli/tg-put-flow-class.html" class="nav-list-link">tg-put-flow-class</a></li><li class="nav-list-item"><a href="/reference/cli/tg-put-kg-core.html" class="nav-list-link">tg-put-kg-core</a></li><li class="nav-list-item"><a href="/reference/cli/tg-remove-library-document.html" class="nav-list-link">tg-remove-library-document</a></li><li class="nav-list-item"><a href="/reference/cli/tg-save-doc-embeds.html" class="nav-list-link">tg-save-doc-embeds</a></li><li class="nav-list-item"><a href="/reference/cli/tg-set-prompt.html" class="nav-list-link">tg-set-prompt</a></li><li class="nav-list-item"><a href="/reference/cli/tg-set-token-costs.html" class="nav-list-link">tg-set-token-costs</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-config.html" class="nav-list-link">tg-show-config</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-flow-classes.html" class="nav-list-link">tg-show-flow-classes</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-flow-state.html" class="nav-list-link">tg-show-flow-state</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-flows.html" class="nav-list-link">tg-show-flows</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-graph.html" class="nav-list-link">tg-show-graph</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-kg-cores.html" class="nav-list-link">tg-show-kg-cores</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-library-documents.html" class="nav-list-link">tg-show-library-documents</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-library-processing.html" class="nav-list-link">tg-show-library-processing</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-processor-state.html" class="nav-list-link">tg-show-processor-state</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-prompts.html" class="nav-list-link">tg-show-prompts</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-token-costs.html" class="nav-list-link">tg-show-token-costs</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-token-rate.html" class="nav-list-link">tg-show-token-rate</a></li><li class="nav-list-item"><a href="/reference/cli/tg-show-tools.html" class="nav-list-link">tg-show-tools</a></li><li class="nav-list-item"><a href="/reference/cli/tg-start-flow.html" class="nav-list-link">tg-start-flow</a></li><li class="nav-list-item"><a href="/reference/cli/tg-start-library-processing.html" class="nav-list-link">tg-start-library-processing</a></li><li class="nav-list-item"><a href="/reference/cli/tg-stop-flow.html" class="nav-list-link">tg-stop-flow</a></li><li class="nav-list-item"><a href="/reference/cli/tg-stop-library-processing.html" class="nav-list-link">tg-stop-library-processing</a></li><li class="nav-list-item"><a href="/reference/cli/tg-unload-kg-core.html" class="nav-list-link">tg-unload-kg-core</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in APIs category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/reference/apis/" class="nav-list-link">APIs</a><ul class="nav-list"><li class="nav-list-item"><a href="/reference/apis/pulsar.html" class="nav-list-link">About Pulsar</a></li><li class="nav-list-item"><a href="/reference/apis/api-agent.html" class="nav-list-link">Agent API</a></li><li class="nav-list-item"><a href="/reference/apis/api-config.html" class="nav-list-link">Config API</a></li><li class="nav-list-item"><a href="/reference/apis/api-core-import-export.html" class="nav-list-link">Core Import/Export API</a></li><li class="nav-list-item"><a href="/reference/apis/api-document-embeddings.html" class="nav-list-link">Document Embeddings API</a></li><li class="nav-list-item"><a href="/reference/apis/api-document-load.html" class="nav-list-link">Document Load API</a></li><li class="nav-list-item"><a href="/reference/apis/api-document-rag.html" class="nav-list-link">Document RAG API</a></li><li class="nav-list-item"><a href="/reference/apis/api-embeddings.html" class="nav-list-link">Embeddings API</a></li><li class="nav-list-item"><a href="/reference/apis/api-entity-contexts.html" class="nav-list-link">Entity Contexts API</a></li><li class="nav-list-item"><a href="/reference/apis/api-flow.html" class="nav-list-link">Flow API</a></li><li class="nav-list-item"><a href="/reference/apis/api-graph-embeddings.html" class="nav-list-link">Graph Embeddings API</a></li><li class="nav-list-item"><a href="/reference/apis/api-graph-rag.html" class="nav-list-link">Graph RAG API</a></li><li class="nav-list-item"><a href="/reference/apis/api-knowledge.html" class="nav-list-link">Knowledge API</a></li><li class="nav-list-item"><a href="/reference/apis/api-librarian.html" class="nav-list-link">Librarian API</a></li><li class="nav-list-item"><a href="/reference/apis/api-metrics.html" class="nav-list-link">Metrics API</a></li><li class="nav-list-item"><a href="/reference/apis/api-prompt.html" class="nav-list-link">Prompt API</a></li><li class="nav-list-item"><a href="/reference/apis/api-text-completion.html" class="nav-list-link">Text Completion API</a></li><li class="nav-list-item"><a href="/reference/apis/api-text-load.html" class="nav-list-link">Text Load API</a></li><li class="nav-list-item"><a href="/reference/apis/api-triples-query.html" class="nav-list-link">Triples Query API</a></li><li class="nav-list-item"><a href="/reference/apis/websocket.html" class="nav-list-link">websocket</a></li></ul></li><li class="nav-list-item"><a href="/reference/containers.html" class="nav-list-link">Containers</a></li><li class="nav-list-item"><a href="/reference/python-packages.html" class="nav-list-link">Python packages</a></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Extending TrustGraph category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/reference/extending/" class="nav-list-link">Extending TrustGraph</a><ul class="nav-list"><li class="nav-list-item"><a href="/reference/extending/async-processor.html" class="nav-list-link">AsyncProcessor</a></li><li class="nav-list-item"><a href="/reference/extending/flow-processor.html" class="nav-list-link">FlowProcessor</a></li><li class="nav-list-item"><a href="/reference/extending/service-base-classes.html" class="nav-list-link">Service Base Classes</a></li><li class="nav-list-item"><a href="/reference/extending/flow-specifications.html" class="nav-list-link">Flow Specifications</a></li></ul></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Community category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/community/" class="nav-list-link">Community</a><ul class="nav-list"><li class="nav-list-item"><a href="/community/changelog.html" class="nav-list-link">Changelog</a></li><li class="nav-list-item"><a href="/community/code-of-conduct.html" class="nav-list-link">Code of Conduct</a></li><li class="nav-list-item"><a href="/community/contributing.html" class="nav-list-link">Contributing</a></li><li class="nav-list-item"><a href="/community/developer.html" class="nav-list-link">Developer's Guide</a></li><li class="nav-list-item"><a href="/community/roadmap.html" class="nav-list-link">Roadmap</a></li><li class="nav-list-item"><a href="/community/support.html" class="nav-list-link">Support</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Examples category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/examples/" class="nav-list-link">Examples</a><ul class="nav-list"><li class="nav-list-item"><a href="/examples/integrations/" class="nav-list-link">Integrations</a></li><li class="nav-list-item"><a href="/examples/sample-data/" class="nav-list-link">Sample Data</a></li><li class="nav-list-item"><a href="/examples/tutorials/" class="nav-list-link">Tutorials</a></li></ul></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Advanced Topics category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/advanced/" class="nav-list-link">Advanced Topics</a><ul class="nav-list"><li class="nav-list-item"><a href="/advanced/backup-restore.html" class="nav-list-link">Backup & Restore</a></li><li class="nav-list-item"><a href="/advanced/clustering.html" class="nav-list-link">Clustering</a></li><li class="nav-list-item"><a href="/advanced/custom-algorithms.html" class="nav-list-link">Custom Algorithms</a></li><li class="nav-list-item"><a href="/advanced/disaster-recovery.html" class="nav-list-link">Disaster Recovery</a></li><li class="nav-list-item"><button class="nav-list-expander btn-reset" aria-label="toggle items in Extending TrustGraph category" aria-pressed="false">
<svg viewBox="0 0 24 24" aria-hidden="true"><use xlink:href="#svg-arrow-right"></use></svg>
</button><a href="/advanced/extending-trustgraph.html" class="nav-list-link">Extending TrustGraph</a><ul class="nav-list"><li class="nav-list-item"><a href="/reference/extending/async-processor.html" class="nav-list-link">AsyncProcessor</a></li><li class="nav-list-item"><a href="/reference/extending/flow-processor.html" class="nav-list-link">FlowProcessor</a></li><li class="nav-list-item"><a href="/reference/extending/service-base-classes.html" class="nav-list-link">Service Base Classes</a></li><li class="nav-list-item"><a href="/reference/extending/flow-specifications.html" class="nav-list-link">Flow Specifications</a></li></ul></li><li class="nav-list-item"><a href="/advanced/performance-tuning.html" class="nav-list-link">Performance Tuning</a></li></ul></li></ul></li></ul>
</nav>
<footer class="site-footer">
This site uses <a href="https://github.com/just-the-docs/just-the-docs">Just the Docs</a>, a documentation theme for Jekyll.
</footer>
</div>
<div class="main" id="top">
<div id="main-header" class="main-header">
<div class="search" role="search">
<div class="search-input-wrap">
<input type="text" id="search-input" class="search-input" tabindex="0" placeholder="Search TrustGraph" aria-label="Search TrustGraph" autocomplete="off">
<label for="search-input" class="search-label"><svg viewBox="0 0 24 24" class="search-icon"><use xlink:href="#svg-search"></use></svg></label>
</div>
<div id="search-results" class="search-results"></div>
</div>
<nav aria-label="Auxiliary" class="aux-nav">
<ul class="aux-nav-list">
<li class="aux-nav-list-item">
<a href="//github.com/trustgraph-ai/trustgraph" class="site-button"
target="_blank" rel="noopener noreferrer"
>
TrustGraph on GitHub
</a>
</li>
</ul>
</nav>
</div>
<div class="main-content-wrap">
<nav aria-label="Breadcrumb" class="breadcrumb-nav">
<ol class="breadcrumb-nav-list">
<li class="breadcrumb-nav-list-item"><a href="/">TrustGraph Documentation</a></li>
<li class="breadcrumb-nav-list-item"><a href="/reference/">Reference</a></li>
<li class="breadcrumb-nav-list-item"><a href="/reference/cli/">CLI</a></li>
<li class="breadcrumb-nav-list-item"><span>tg-save-doc-embeds</span></li>
</ol>
</nav>
<div id="main-content" class="main-content">
<main>
<h1 id="tg-save-doc-embeds">
<a href="#tg-save-doc-embeds" class="anchor-heading" aria-labelledby="tg-save-doc-embeds"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> tg-save-doc-embeds
</h1>
<p>Saves document embeddings from TrustGraph processing streams to MessagePack format files.</p>
<h2 id="synopsis">
<a href="#synopsis" class="anchor-heading" aria-labelledby="synopsis"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Synopsis
</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="nt">-o</span> OUTPUT_FILE <span class="o">[</span>options]
</code></pre></div></div>
<h2 id="description">
<a href="#description" class="anchor-heading" aria-labelledby="description"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Description
</h2>
<p>The <code class="language-plaintext highlighter-rouge">tg-save-doc-embeds</code> command connects to TrustGraph’s document embeddings export stream and saves the embeddings to a file in MessagePack format. This is useful for creating backups of document embeddings, exporting data for analysis, or preparing data for migration between systems.</p>
<p>The command should typically be started before document processing begins to capture all embeddings as they are generated.</p>
<h2 id="options">
<a href="#options" class="anchor-heading" aria-labelledby="options"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Options
</h2>
<h3 id="required-arguments">
<a href="#required-arguments" class="anchor-heading" aria-labelledby="required-arguments"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Required Arguments
</h3>
<ul>
<li><code class="language-plaintext highlighter-rouge">-o, --output-file FILE</code>: Output file for saved embeddings</li>
</ul>
<h3 id="optional-arguments">
<a href="#optional-arguments" class="anchor-heading" aria-labelledby="optional-arguments"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Optional Arguments
</h3>
<ul>
<li><code class="language-plaintext highlighter-rouge">-u, --url URL</code>: TrustGraph API URL (default: <code class="language-plaintext highlighter-rouge">$TRUSTGRAPH_API</code> or <code class="language-plaintext highlighter-rouge">http://localhost:8088/</code>)</li>
<li><code class="language-plaintext highlighter-rouge">-f, --flow-id ID</code>: Flow instance ID to monitor (default: <code class="language-plaintext highlighter-rouge">default</code>)</li>
<li><code class="language-plaintext highlighter-rouge">--format FORMAT</code>: Output format - <code class="language-plaintext highlighter-rouge">msgpack</code> or <code class="language-plaintext highlighter-rouge">json</code> (default: <code class="language-plaintext highlighter-rouge">msgpack</code>)</li>
<li><code class="language-plaintext highlighter-rouge">--user USER</code>: Filter by user ID (default: no filter)</li>
<li><code class="language-plaintext highlighter-rouge">--collection COLLECTION</code>: Filter by collection ID (default: no filter)</li>
</ul>
<h2 id="examples">
<a href="#examples" class="anchor-heading" aria-labelledby="examples"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Examples
</h2>
<h3 id="basic-document-embeddings-export">
<a href="#basic-document-embeddings-export" class="anchor-heading" aria-labelledby="basic-document-embeddings-export"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Basic Document Embeddings Export
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="nt">-o</span> document-embeddings.msgpack
</code></pre></div></div>
<h3 id="export-from-specific-flow">
<a href="#export-from-specific-flow" class="anchor-heading" aria-labelledby="export-from-specific-flow"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Export from Specific Flow
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> research-embeddings.msgpack <span class="se">\</span>
<span class="nt">-f</span> <span class="s2">"research-processing-flow"</span>
</code></pre></div></div>
<h3 id="filter-by-user-and-collection">
<a href="#filter-by-user-and-collection" class="anchor-heading" aria-labelledby="filter-by-user-and-collection"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Filter by User and Collection
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> filtered-embeddings.msgpack <span class="se">\</span>
<span class="nt">--user</span> <span class="s2">"research-team"</span> <span class="se">\</span>
<span class="nt">--collection</span> <span class="s2">"research-docs"</span>
</code></pre></div></div>
<h3 id="export-to-json-format">
<a href="#export-to-json-format" class="anchor-heading" aria-labelledby="export-to-json-format"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Export to JSON Format
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> embeddings.json <span class="se">\</span>
<span class="nt">--format</span> json
</code></pre></div></div>
<h3 id="production-backup">
<a href="#production-backup" class="anchor-heading" aria-labelledby="production-backup"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Production Backup
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"backup-</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d-%H%M%S<span class="si">)</span><span class="s2">.msgpack"</span> <span class="se">\</span>
<span class="nt">-u</span> https://production-api.company.com/ <span class="se">\</span>
<span class="nt">-f</span> <span class="s2">"production-flow"</span>
</code></pre></div></div>
<h2 id="output-format">
<a href="#output-format" class="anchor-heading" aria-labelledby="output-format"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Output Format
</h2>
<h3 id="messagepack-structure">
<a href="#messagepack-structure" class="anchor-heading" aria-labelledby="messagepack-structure"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> MessagePack Structure
</h3>
<p>Document embeddings are saved as MessagePack records:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="s2">"de"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"m"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"i"</span><span class="p">:</span><span class="w"> </span><span class="s2">"document-id"</span><span class="p">,</span><span class="w">
</span><span class="nl">"m"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="nl">"metadata"</span><span class="p">:</span><span class="w"> </span><span class="s2">"objects"</span><span class="p">}],</span><span class="w">
</span><span class="nl">"u"</span><span class="p">:</span><span class="w"> </span><span class="s2">"user-id"</span><span class="p">,</span><span class="w">
</span><span class="nl">"c"</span><span class="p">:</span><span class="w"> </span><span class="s2">"collection-id"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"c"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"c"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text chunk content"</span><span class="p">,</span><span class="w">
</span><span class="nl">"v"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="mf">0.1</span><span class="p">,</span><span class="w"> </span><span class="mf">0.2</span><span class="p">,</span><span class="w"> </span><span class="mf">0.3</span><span class="p">,</span><span class="w"> </span><span class="err">...</span><span class="p">]</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span></code></pre></div></div>
<h3 id="components">
<a href="#components" class="anchor-heading" aria-labelledby="components"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Components
</h3>
<ul>
<li><strong>Record Type</strong>: <code class="language-plaintext highlighter-rouge">"de"</code> indicates document embeddings</li>
<li><strong>Metadata</strong> (<code class="language-plaintext highlighter-rouge">m</code>): Document information and context</li>
<li><strong>Chunks</strong> (<code class="language-plaintext highlighter-rouge">c</code>): Text chunks with their vector embeddings</li>
</ul>
<h2 id="use-cases">
<a href="#use-cases" class="anchor-heading" aria-labelledby="use-cases"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Use Cases
</h2>
<h3 id="backup-creation">
<a href="#backup-creation" class="anchor-heading" aria-labelledby="backup-creation"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Backup Creation
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Create regular backups of document embeddings</span>
create_embeddings_backup<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">backup_dir</span><span class="o">=</span><span class="s2">"embeddings-backups"</span>
<span class="nb">local </span><span class="nv">timestamp</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d_%H%M%S<span class="si">)</span>
<span class="nb">local </span><span class="nv">backup_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$backup_dir</span><span class="s2">/embeddings-</span><span class="nv">$timestamp</span><span class="s2">.msgpack"</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> <span class="s2">"</span><span class="nv">$backup_dir</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Creating embeddings backup: </span><span class="nv">$backup_file</span><span class="s2">"</span>
<span class="c"># Start backup process</span>
tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"</span><span class="nv">$backup_file</span><span class="s2">"</span> &
<span class="nv">save_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="nb">echo</span> <span class="s2">"Backup process started (PID: </span><span class="nv">$save_pid</span><span class="s2">)"</span>
<span class="nb">echo</span> <span class="s2">"To stop: kill </span><span class="nv">$save_pid</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Backup file: </span><span class="nv">$backup_file</span><span class="s2">"</span>
<span class="c"># Optionally wait for a specific duration</span>
<span class="c"># sleep 3600 # Run for 1 hour</span>
<span class="c"># kill $save_pid</span>
<span class="o">}</span>
<span class="c"># Create backup</span>
create_embeddings_backup
</code></pre></div></div>
<h3 id="data-migration-preparation">
<a href="#data-migration-preparation" class="anchor-heading" aria-labelledby="data-migration-preparation"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Data Migration Preparation
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Prepare embeddings for migration</span>
prepare_migration_data<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">source_env</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">collection</span><span class="o">=</span><span class="s2">"</span><span class="nv">$2</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">migration_file</span><span class="o">=</span><span class="s2">"migration-</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d<span class="si">)</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Preparing migration data from: </span><span class="nv">$source_env</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Collection: </span><span class="nv">$collection</span><span class="s2">"</span>
<span class="c"># Export embeddings from source</span>
tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"</span><span class="nv">$migration_file</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">-u</span> <span class="s2">"http://</span><span class="nv">$source_env</span><span class="s2">:8088/"</span> <span class="se">\</span>
<span class="nt">--collection</span> <span class="s2">"</span><span class="nv">$collection</span><span class="s2">"</span> &
<span class="nv">export_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="c"># Let it run for specified time to capture data</span>
<span class="nb">echo</span> <span class="s2">"Capturing embeddings for migration..."</span>
<span class="nb">echo</span> <span class="s2">"Process PID: </span><span class="nv">$export_pid</span><span class="s2">"</span>
<span class="c"># In practice, you'd run this for the duration needed</span>
<span class="c"># sleep 1800 # 30 minutes</span>
<span class="c"># kill $export_pid</span>
<span class="nb">echo</span> <span class="s2">"Migration data will be saved to: </span><span class="nv">$migration_file</span><span class="s2">"</span>
<span class="o">}</span>
<span class="c"># Prepare migration from dev to production</span>
prepare_migration_data <span class="s2">"dev-server"</span> <span class="s2">"processed-docs"</span>
</code></pre></div></div>
<h3 id="continuous-export">
<a href="#continuous-export" class="anchor-heading" aria-labelledby="continuous-export"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Continuous Export
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Continuous embeddings export with rotation</span>
continuous_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">output_dir</span><span class="o">=</span><span class="s2">"continuous-exports"</span>
<span class="nb">local </span><span class="nv">rotation_hours</span><span class="o">=</span>24
<span class="nb">local </span><span class="nv">file_prefix</span><span class="o">=</span><span class="s2">"embeddings"</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> <span class="s2">"</span><span class="nv">$output_dir</span><span class="s2">"</span>
<span class="k">while </span><span class="nb">true</span><span class="p">;</span> <span class="k">do
</span><span class="nv">timestamp</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d_%H%M%S<span class="si">)</span>
<span class="nv">output_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$output_dir</span><span class="s2">/</span><span class="k">${</span><span class="nv">file_prefix</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="nv">timestamp</span><span class="k">}</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Starting export to: </span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="c"># Start export for specified duration</span>
<span class="nb">timeout</span> <span class="k">${</span><span class="nv">rotation_hours</span><span class="k">}</span>h tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="c"># Compress completed file</span>
<span class="nb">gzip</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Export completed and compressed: </span><span class="k">${</span><span class="nv">output_file</span><span class="k">}</span><span class="s2">.gz"</span>
<span class="c"># Optional: clean up old files</span>
find <span class="s2">"</span><span class="nv">$output_dir</span><span class="s2">"</span> <span class="nt">-name</span> <span class="s2">"*.msgpack.gz"</span> <span class="nt">-mtime</span> +30 <span class="nt">-delete</span>
<span class="c"># Brief pause before next rotation</span>
<span class="nb">sleep </span>60
<span class="k">done</span>
<span class="o">}</span>
<span class="c"># Start continuous export (run in background)</span>
continuous_export &
</code></pre></div></div>
<h3 id="analysis-and-research">
<a href="#analysis-and-research" class="anchor-heading" aria-labelledby="analysis-and-research"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Analysis and Research
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Export embeddings for research analysis</span>
export_for_research<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">research_topic</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">output_file</span><span class="o">=</span><span class="s2">"research-</span><span class="k">${</span><span class="nv">research_topic</span><span class="k">}</span><span class="s2">-</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d<span class="si">)</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Exporting embeddings for research: </span><span class="nv">$research_topic</span><span class="s2">"</span>
<span class="c"># Start export with filtering</span>
tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--collection</span> <span class="s2">"</span><span class="nv">$research_topic</span><span class="s2">"</span> &
<span class="nv">export_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="nb">echo</span> <span class="s2">"Research export started (PID: </span><span class="nv">$export_pid</span><span class="s2">)"</span>
<span class="nb">echo</span> <span class="s2">"Output: </span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="c"># Create analysis script</span>
<span class="nb">cat</span> <span class="o">></span> <span class="s2">"analyze-</span><span class="k">${</span><span class="nv">research_topic</span><span class="k">}</span><span class="s2">.sh"</span> <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
#!/bin/bash
# Analysis script for </span><span class="nv">$research_topic</span><span class="sh"> embeddings
echo "Analyzing </span><span class="nv">$research_topic</span><span class="sh"> embeddings..."
# Basic statistics
echo "=== Basic Statistics ==="
tg-dump-msgpack -i "</span><span class="nv">$output_file</span><span class="sh">" --summary
# Detailed analysis
echo "=== Detailed Analysis ==="
tg-dump-msgpack -i "</span><span class="nv">$output_file</span><span class="sh">" | head -10
echo "Analysis complete for </span><span class="nv">$research_topic</span><span class="sh">"
</span><span class="no">EOF
</span> <span class="nb">chmod</span> +x <span class="s2">"analyze-</span><span class="k">${</span><span class="nv">research_topic</span><span class="k">}</span><span class="s2">.sh"</span>
<span class="nb">echo</span> <span class="s2">"Analysis script created: analyze-</span><span class="k">${</span><span class="nv">research_topic</span><span class="k">}</span><span class="s2">.sh"</span>
<span class="o">}</span>
<span class="c"># Export for different research topics</span>
export_for_research <span class="s2">"cybersecurity"</span>
export_for_research <span class="s2">"climate-change"</span>
</code></pre></div></div>
<h2 id="advanced-usage">
<a href="#advanced-usage" class="anchor-heading" aria-labelledby="advanced-usage"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Advanced Usage
</h2>
<h3 id="selective-export">
<a href="#selective-export" class="anchor-heading" aria-labelledby="selective-export"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Selective Export
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Export embeddings with multiple filters</span>
selective_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local users</span><span class="o">=(</span><span class="s2">"user1"</span> <span class="s2">"user2"</span> <span class="s2">"user3"</span><span class="o">)</span>
<span class="nb">local </span><span class="nv">collections</span><span class="o">=(</span><span class="s2">"docs1"</span> <span class="s2">"docs2"</span><span class="o">)</span>
<span class="k">for </span>user <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="nv">users</span><span class="p">[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
for </span>collection <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="nv">collections</span><span class="p">[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
</span><span class="nv">output_file</span><span class="o">=</span><span class="s2">"embeddings-</span><span class="k">${</span><span class="nv">user</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="nv">collection</span><span class="k">}</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Exporting for user: </span><span class="nv">$user</span><span class="s2">, collection: </span><span class="nv">$collection</span><span class="s2">"</span>
tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--user</span> <span class="s2">"</span><span class="nv">$user</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--collection</span> <span class="s2">"</span><span class="nv">$collection</span><span class="s2">"</span> &
<span class="c"># Store PID for later management</span>
<span class="nb">echo</span> <span class="nv">$!</span> <span class="o">></span> <span class="s2">"</span><span class="k">${</span><span class="nv">output_file</span><span class="k">}</span><span class="s2">.pid"</span>
<span class="k">done
done
</span><span class="nb">echo</span> <span class="s2">"All selective exports started"</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="monitoring-and-statistics">
<a href="#monitoring-and-statistics" class="anchor-heading" aria-labelledby="monitoring-and-statistics"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Monitoring and Statistics
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Monitor export progress with statistics</span>
monitor_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">output_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">pid_file</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">output_file</span><span class="k">}</span><span class="s2">.pid"</span>
<span class="k">if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$pid_file</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"PID file not found: </span><span class="nv">$pid_file</span><span class="s2">"</span>
<span class="k">return </span>1
<span class="k">fi
</span><span class="nb">local </span><span class="nv">export_pid</span><span class="o">=</span><span class="si">$(</span><span class="nb">cat</span> <span class="s2">"</span><span class="nv">$pid_file</span><span class="s2">"</span><span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"Monitoring export (PID: </span><span class="nv">$export_pid</span><span class="s2">)..."</span>
<span class="nb">echo</span> <span class="s2">"Output file: </span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="k">while </span><span class="nb">kill</span> <span class="nt">-0</span> <span class="s2">"</span><span class="nv">$export_pid</span><span class="s2">"</span> 2>/dev/null<span class="p">;</span> <span class="k">do
if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nv">file_size</span><span class="o">=</span><span class="si">$(</span><span class="nb">stat</span> <span class="nt">-c</span>%s <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> 2>/dev/null <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"0"</span><span class="si">)</span>
<span class="nv">human_size</span><span class="o">=</span><span class="si">$(</span><span class="nb">numfmt</span> <span class="nt">--to</span><span class="o">=</span>iec-i <span class="nt">--suffix</span><span class="o">=</span>B <span class="s2">"</span><span class="nv">$file_size</span><span class="s2">"</span><span class="si">)</span>
<span class="c"># Try to count embeddings</span>
<span class="nv">embedding_count</span><span class="o">=</span><span class="si">$(</span>tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> 2>/dev/null | <span class="nb">grep</span> <span class="nt">-c</span> <span class="s1">'^\["de"'</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"0"</span><span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"File size: </span><span class="nv">$human_size</span><span class="s2">, Embeddings: </span><span class="nv">$embedding_count</span><span class="s2">"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"Output file not yet created..."</span>
<span class="k">fi
</span><span class="nb">sleep </span>30
<span class="k">done
</span><span class="nb">echo</span> <span class="s2">"Export process completed"</span>
<span class="nb">rm</span> <span class="s2">"</span><span class="nv">$pid_file</span><span class="s2">"</span>
<span class="o">}</span>
<span class="c"># Start export and monitor</span>
tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"monitored-export.msgpack"</span> &
<span class="nb">echo</span> <span class="nv">$!</span> <span class="o">></span> <span class="s2">"monitored-export.msgpack.pid"</span>
monitor_export <span class="s2">"monitored-export.msgpack"</span>
</code></pre></div></div>
<h3 id="export-validation">
<a href="#export-validation" class="anchor-heading" aria-labelledby="export-validation"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Export Validation
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Validate exported embeddings</span>
validate_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">export_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Validating export file: </span><span class="nv">$export_file</span><span class="s2">"</span>
<span class="c"># Check file exists and has content</span>
<span class="k">if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-s</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"✗ Export file is empty or missing"</span>
<span class="k">return </span>1
<span class="k">fi</span>
<span class="c"># Check MessagePack format</span>
<span class="k">if </span>tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> <span class="nt">--summary</span> <span class="o">></span> /dev/null 2>&1<span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"✓ Valid MessagePack format"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"✗ Invalid MessagePack format"</span>
<span class="k">return </span>1
<span class="k">fi</span>
<span class="c"># Check for document embeddings</span>
<span class="nv">embedding_count</span><span class="o">=</span><span class="si">$(</span>tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> | <span class="nb">grep</span> <span class="nt">-c</span> <span class="s1">'^\["de"'</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"0"</span><span class="si">)</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$embedding_count</span><span class="s2">"</span> <span class="nt">-gt</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"✓ Contains </span><span class="nv">$embedding_count</span><span class="s2"> document embeddings"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"✗ No document embeddings found"</span>
<span class="k">return </span>1
<span class="k">fi</span>
<span class="c"># Get vector dimension information</span>
<span class="nv">summary</span><span class="o">=</span><span class="si">$(</span>tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> <span class="nt">--summary</span><span class="si">)</span>
<span class="k">if </span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$summary</span><span class="s2">"</span> | <span class="nb">grep</span> <span class="nt">-q</span> <span class="s2">"Vector dimension:"</span><span class="p">;</span> <span class="k">then
</span><span class="nv">dimension</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$summary</span><span class="s2">"</span> | <span class="nb">grep</span> <span class="s2">"Vector dimension:"</span> | <span class="nb">awk</span> <span class="s1">'{print $3}'</span><span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"✓ Vector dimension: </span><span class="nv">$dimension</span><span class="s2">"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"⚠ Could not determine vector dimension"</span>
<span class="k">fi
</span><span class="nb">echo</span> <span class="s2">"Validation completed successfully"</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="export-scheduling">
<a href="#export-scheduling" class="anchor-heading" aria-labelledby="export-scheduling"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Export Scheduling
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Scheduled export with cron-like functionality</span>
schedule_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">schedule</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="c"># e.g., "daily", "hourly", "weekly"</span>
<span class="nb">local </span><span class="nv">output_prefix</span><span class="o">=</span><span class="s2">"</span><span class="nv">$2</span><span class="s2">"</span>
<span class="k">case</span> <span class="s2">"</span><span class="nv">$schedule</span><span class="s2">"</span> <span class="k">in</span>
<span class="s2">"hourly"</span><span class="p">)</span>
<span class="nv">interval</span><span class="o">=</span>3600
<span class="p">;;</span>
<span class="s2">"daily"</span><span class="p">)</span>
<span class="nv">interval</span><span class="o">=</span>86400
<span class="p">;;</span>
<span class="s2">"weekly"</span><span class="p">)</span>
<span class="nv">interval</span><span class="o">=</span>604800
<span class="p">;;</span>
<span class="k">*</span><span class="p">)</span>
<span class="nb">echo</span> <span class="s2">"Invalid schedule: </span><span class="nv">$schedule</span><span class="s2">"</span>
<span class="k">return </span>1
<span class="p">;;</span>
<span class="k">esac</span>
<span class="nb">echo</span> <span class="s2">"Starting </span><span class="nv">$schedule</span><span class="s2"> exports with prefix: </span><span class="nv">$output_prefix</span><span class="s2">"</span>
<span class="k">while </span><span class="nb">true</span><span class="p">;</span> <span class="k">do
</span><span class="nv">timestamp</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d_%H%M%S<span class="si">)</span>
<span class="nv">output_file</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">output_prefix</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="nv">timestamp</span><span class="k">}</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Starting scheduled export: </span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="c"># Run export for the scheduled interval</span>
<span class="nb">timeout</span> <span class="k">${</span><span class="nv">interval</span><span class="k">}</span>s tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="c"># Validate and compress</span>
<span class="k">if </span>validate_export <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span><span class="p">;</span> <span class="k">then
</span><span class="nb">gzip</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"✓ Export completed and compressed: </span><span class="k">${</span><span class="nv">output_file</span><span class="k">}</span><span class="s2">.gz"</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"✗ Export validation failed: </span><span class="nv">$output_file</span><span class="s2">"</span>
<span class="nb">mv</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> <span class="s2">"</span><span class="k">${</span><span class="nv">output_file</span><span class="k">}</span><span class="s2">.failed"</span>
<span class="k">fi</span>
<span class="c"># Brief pause before next cycle</span>
<span class="nb">sleep </span>60
<span class="k">done</span>
<span class="o">}</span>
<span class="c"># Start daily scheduled exports</span>
schedule_export <span class="s2">"daily"</span> <span class="s2">"daily-embeddings"</span> &
</code></pre></div></div>
<h2 id="performance-considerations">
<a href="#performance-considerations" class="anchor-heading" aria-labelledby="performance-considerations"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Performance Considerations
</h2>
<h3 id="memory-management">
<a href="#memory-management" class="anchor-heading" aria-labelledby="memory-management"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Memory Management
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Monitor memory usage during export</span>
monitor_memory_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">output_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="c"># Start export</span>
tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> &
<span class="nv">export_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="nb">echo</span> <span class="s2">"Monitoring memory usage for export (PID: </span><span class="nv">$export_pid</span><span class="s2">)..."</span>
<span class="k">while </span><span class="nb">kill</span> <span class="nt">-0</span> <span class="s2">"</span><span class="nv">$export_pid</span><span class="s2">"</span> 2>/dev/null<span class="p">;</span> <span class="k">do
</span><span class="nv">memory_usage</span><span class="o">=</span><span class="si">$(</span>ps <span class="nt">-p</span> <span class="s2">"</span><span class="nv">$export_pid</span><span class="s2">"</span> <span class="nt">-o</span> <span class="nv">rss</span><span class="o">=</span> 2>/dev/null | <span class="nb">awk</span> <span class="s1">'{print $1/1024}'</span><span class="si">)</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-n</span> <span class="s2">"</span><span class="nv">$memory_usage</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Memory usage: </span><span class="k">${</span><span class="nv">memory_usage</span><span class="k">}</span><span class="s2">MB"</span>
<span class="k">fi
</span><span class="nb">sleep </span>10
<span class="k">done
</span><span class="nb">echo</span> <span class="s2">"Export completed"</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="network-optimization">
<a href="#network-optimization" class="anchor-heading" aria-labelledby="network-optimization"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Network Optimization
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Optimize for network conditions</span>
network_optimized_export<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">output_file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">api_url</span><span class="o">=</span><span class="s2">"</span><span class="nv">$2</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"Starting network-optimized export..."</span>
<span class="c"># Use compression and buffering</span>
tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"</span><span class="nv">$output_file</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">-u</span> <span class="s2">"</span><span class="nv">$api_url</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--format</span> msgpack & <span class="c"># MessagePack is more compact than JSON</span>
<span class="nv">export_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="c"># Monitor network usage</span>
<span class="nb">echo</span> <span class="s2">"Monitoring export (PID: </span><span class="nv">$export_pid</span><span class="s2">)..."</span>
<span class="k">while </span><span class="nb">kill</span> <span class="nt">-0</span> <span class="s2">"</span><span class="nv">$export_pid</span><span class="s2">"</span> 2>/dev/null<span class="p">;</span> <span class="k">do</span>
<span class="c"># Monitor network connections</span>
<span class="nv">connections</span><span class="o">=</span><span class="si">$(</span>netstat <span class="nt">-an</span> | <span class="nb">grep</span> <span class="s2">":8088"</span> | <span class="nb">wc</span> <span class="nt">-l</span><span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"Active connections: </span><span class="nv">$connections</span><span class="s2">"</span>
<span class="nb">sleep </span>30
<span class="k">done</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="error-handling">
<a href="#error-handling" class="anchor-heading" aria-labelledby="error-handling"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Error Handling
</h2>
<h3 id="connection-issues">
<a href="#connection-issues" class="anchor-heading" aria-labelledby="connection-issues"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Connection Issues
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Exception: WebSocket connection failed
</code></pre></div></div>
<p><strong>Solution</strong>: Check API URL and ensure TrustGraph WebSocket service is running.</p>
<h3 id="disk-space-issues">
<a href="#disk-space-issues" class="anchor-heading" aria-labelledby="disk-space-issues"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Disk Space Issues
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Exception: No space left on device
</code></pre></div></div>
<p><strong>Solution</strong>: Free up disk space or use a different output location.</p>
<h3 id="permission-errors">
<a href="#permission-errors" class="anchor-heading" aria-labelledby="permission-errors"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Permission Errors
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Exception: Permission denied
</code></pre></div></div>
<p><strong>Solution</strong>: Check write permissions for the output file location.</p>
<h3 id="memory-issues">
<a href="#memory-issues" class="anchor-heading" aria-labelledby="memory-issues"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Memory Issues
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MemoryError: Unable to allocate memory
</code></pre></div></div>
<p><strong>Solution</strong>: Monitor memory usage and consider using smaller export windows.</p>
<h2 id="integration-with-other-commands">
<a href="#integration-with-other-commands" class="anchor-heading" aria-labelledby="integration-with-other-commands"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Integration with Other Commands
</h2>
<h3 id="complete-backup-workflow">
<a href="#complete-backup-workflow" class="anchor-heading" aria-labelledby="complete-backup-workflow"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Complete Backup Workflow
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Complete backup and restore workflow</span>
backup_restore_workflow<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">backup_file</span><span class="o">=</span><span class="s2">"embeddings-backup.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"=== Backup Phase ==="</span>
<span class="c"># Create backup</span>
tg-save-doc-embeds <span class="nt">-o</span> <span class="s2">"</span><span class="nv">$backup_file</span><span class="s2">"</span> &
<span class="nv">backup_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="c"># Let it run for a while</span>
<span class="nb">sleep </span>300 <span class="c"># 5 minutes</span>
<span class="nb">kill</span> <span class="nv">$backup_pid</span>
<span class="nb">echo</span> <span class="s2">"Backup created: </span><span class="nv">$backup_file</span><span class="s2">"</span>
<span class="c"># Validate backup</span>
validate_export <span class="s2">"</span><span class="nv">$backup_file</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"=== Restore Phase ==="</span>
<span class="c"># Restore from backup (to different collection)</span>
tg-load-doc-embeds <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$backup_file</span><span class="s2">"</span> <span class="nt">--collection</span> <span class="s2">"restored"</span>
<span class="nb">echo</span> <span class="s2">"Backup and restore workflow completed"</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="analysis-pipeline">
<a href="#analysis-pipeline" class="anchor-heading" aria-labelledby="analysis-pipeline"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Analysis Pipeline
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Export and analyze embeddings</span>
export_analyze_pipeline<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">topic</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">local </span><span class="nv">export_file</span><span class="o">=</span><span class="s2">"analysis-</span><span class="k">${</span><span class="nv">topic</span><span class="k">}</span><span class="s2">.msgpack"</span>
<span class="nb">echo</span> <span class="s2">"Starting export and analysis pipeline for: </span><span class="nv">$topic</span><span class="s2">"</span>
<span class="c"># Export embeddings</span>
tg-save-doc-embeds <span class="se">\</span>
<span class="nt">-o</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> <span class="se">\</span>
<span class="nt">--collection</span> <span class="s2">"</span><span class="nv">$topic</span><span class="s2">"</span> &
<span class="nv">export_pid</span><span class="o">=</span><span class="nv">$!</span>
<span class="c"># Run for analysis duration</span>
<span class="nb">sleep </span>600 <span class="c"># 10 minutes</span>
<span class="nb">kill</span> <span class="nv">$export_pid</span>
<span class="c"># Analyze exported data</span>
<span class="nb">echo</span> <span class="s2">"Analyzing exported embeddings..."</span>
tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> <span class="nt">--summary</span>
<span class="c"># Count embeddings by user</span>
<span class="nb">echo</span> <span class="s2">"Embeddings by user:"</span>
tg-dump-msgpack <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$export_file</span><span class="s2">"</span> | <span class="se">\</span>
jq <span class="nt">-r</span> <span class="s1">'.[1].m.u'</span> | <span class="se">\</span>
<span class="nb">sort</span> | <span class="nb">uniq</span> <span class="nt">-c</span>
<span class="nb">echo</span> <span class="s2">"Analysis pipeline completed"</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="environment-variables">
<a href="#environment-variables" class="anchor-heading" aria-labelledby="environment-variables"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Environment Variables
</h2>
<ul>
<li><code class="language-plaintext highlighter-rouge">TRUSTGRAPH_API</code>: Default API URL</li>
</ul>
<h2 id="related-commands">
<a href="#related-commands" class="anchor-heading" aria-labelledby="related-commands"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Related Commands
</h2>
<ul>
<li><a href="tg-load-doc-embeds"><code class="language-plaintext highlighter-rouge">tg-load-doc-embeds</code></a> - Load document embeddings from files</li>
<li><a href="tg-dump-msgpack"><code class="language-plaintext highlighter-rouge">tg-dump-msgpack</code></a> - Analyze MessagePack files</li>
<li><a href="tg-show-flows"><code class="language-plaintext highlighter-rouge">tg-show-flows</code></a> - List available flows for monitoring</li>
</ul>
<h2 id="api-integration">
<a href="#api-integration" class="anchor-heading" aria-labelledby="api-integration"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> API Integration
</h2>
<p>This command uses TrustGraph’s WebSocket API for document embeddings export, specifically the <code class="language-plaintext highlighter-rouge">/api/v1/flow/{flow-id}/export/document-embeddings</code> endpoint.</p>
<h2 id="best-practices">
<a href="#best-practices" class="anchor-heading" aria-labelledby="best-practices"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Best Practices
</h2>
<ol>
<li><strong>Start Early</strong>: Begin export before processing starts to capture all data</li>
<li><strong>Monitoring</strong>: Monitor export progress and file sizes</li>
<li><strong>Validation</strong>: Always validate exported files</li>
<li><strong>Compression</strong>: Use compression for long-term storage</li>
<li><strong>Rotation</strong>: Implement file rotation for continuous exports</li>
<li><strong>Backup</strong>: Keep multiple backup copies in different locations</li>
<li><strong>Documentation</strong>: Document export schedules and procedures</li>
</ol>
<h2 id="troubleshooting">
<a href="#troubleshooting" class="anchor-heading" aria-labelledby="troubleshooting"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Troubleshooting
</h2>
<h3 id="no-data-captured">
<a href="#no-data-captured" class="anchor-heading" aria-labelledby="no-data-captured"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> No Data Captured
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Check if processing is generating embeddings</span>
tg-show-flows | <span class="nb">grep </span>processing
<span class="c"># Verify WebSocket connection</span>
netstat <span class="nt">-an</span> | <span class="nb">grep</span> :8088
</code></pre></div></div>
<h3 id="large-file-issues">
<a href="#large-file-issues" class="anchor-heading" aria-labelledby="large-file-issues"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Large File Issues
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Monitor file growth</span>
watch <span class="nt">-n</span> 5 <span class="s1">'ls -lh *.msgpack'</span>
<span class="c"># Check available disk space</span>
<span class="nb">df</span> <span class="nt">-h</span>
</code></pre></div></div>
<h3 id="process-management">
<a href="#process-management" class="anchor-heading" aria-labelledby="process-management"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Process Management
</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># List running export processes</span>
ps aux | <span class="nb">grep </span>tg-save-doc-embeds
<span class="c"># Kill stuck processes</span>
pkill <span class="nt">-f</span> tg-save-doc-embeds
</code></pre></div></div>
</main>
<hr>
<footer>
<p class="text-small text-grey-dk-100 mb-0">Copyright © 2024-2025 TrustGraph.ai. Distributed under an <a href="https://github.com/trustgraph-ai/trustgraph/blob/master/LICENSE">Apache 2 license.</a></p>
<div class="d-flex mt-2">
<p class="text-small text-grey-dk-000 mb-0">
<a href="https://github.com/trustgraph-ai/trustgraph-ai.github.io/tree/main/reference/cli/tg-save-doc-embeds.md" id="edit-this-page">Edit this page on GitHub.</a>
</p>
</div>
</footer>
</div>
</div>
<div class="search-overlay"></div>
</div>
<script src="https://cdn.jsdelivr.net/npm/mermaid@9.1.3/dist/mermaid.min.js"></script>
<script>
var config = {}
;
mermaid.initialize(config);
window.mermaid.init(undefined, document.querySelectorAll('.language-mermaid'));
</script>
</body>
</html>
Summary:"
Variable Naming
# Use descriptive variable names
tg-set-prompt \
--id "descriptive-vars" \
--prompt "Analyze data from for trends"
# Group related variables
tg-set-prompt \
--id "grouped-vars" \
--prompt "Compare vs using "
Environment Variables
TRUSTGRAPH_URL: Default API URL
Related Commands
tg-show-prompts- Display configured promptstg-invoke-prompt- Use prompt templatestg-invoke-document-rag- Document-based AI queries
API Integration
This command uses the Config API to store prompt templates and system prompts in TrustGraph’s configuration system.
Best Practices
- Clear Templates: Write clear, specific prompt templates
- Variable Names: Use descriptive variable names
- Response Types: Choose appropriate response types for your use case
- Schema Validation: Always validate JSON schemas before setting
- Version Control: Consider versioning important templates
- Testing: Test templates thoroughly with various inputs
- Documentation: Document template variables and expected usage
Troubleshooting
Template Not Working
# Check template exists
tg-show-prompts | grep "template-id"
# Verify variable names match
tg-invoke-prompt template-id var1="test" var2="test"
JSON Schema Errors
# Validate schema separately
echo '{"type": "object"}' | jq .
# Test with simple schema first
tg-set-prompt --id "test" --prompt "test" --response "json" --schema '{"type": "string"}'
System Prompt Issues
# Check current system prompt
tg-show-prompts | grep -A5 "System prompt"
# Reset if needed
tg-set-prompt --system "Default system prompt"