Merlin API Reference
Version 2.4.7The DragonFire Merlin API provides advanced recursive harmonic data compression and processing capabilities. This API enables your applications to compress and process data with unprecedented efficiency using mathematically optimal patterns.
Overview
The Merlin API provides recursive harmonic data compression and intelligent data processing capabilities. By leveraging mathematical constants and patterns found in nature, Merlin achieves compression ratios that exceed traditional algorithms while preserving data integrity.
Key Features
- Recursive Compression: Multi-pass recursive compression using harmonic patterns
- Adaptive Algorithms: Self-optimizing compression based on data characteristics
- Pattern Recognition: Identifies and optimizes for mathematical patterns in data
- Streaming Support: Efficient compression of streaming data
- Lossless and Lossy Modes: Configurable precision for different use cases
Compression Architecture
Client SDK Requirements
To use the Merlin API, you'll need the DragonFire Merlin Client SDK:
# Using npm
npm install @dragonfire/merlin-client
# Using yarn
yarn add @dragonfire/merlin-client
Initialization
Before using the Merlin API, you need to initialize a client instance with a connection to the DragonFire service.
Creating a Merlin Client
MerlinClient constructor
Creates a new client for interacting with the DragonFire Merlin service.
Syntax
constructor(dragonfire: DragonFireClient, options?: MerlinClientOptions)
Parameters
Name | Type | Description |
---|---|---|
dragonfire | DragonFireClient | An initialized and connected DragonFire client instance. |
options | MerlinClientOptions | Optional. Configuration options for the Merlin client. |
Example
import { DragonFireClient } from '@dragonfire/client';
import { MerlinClient } from '@dragonfire/merlin-client';
// Initialize DragonFire client
const dragonfire = new DragonFireClient({
apiKey: 'YOUR_API_KEY',
region: 'us-west'
});
await dragonfire.connect();
// Initialize Merlin client
const merlin = new MerlinClient(dragonfire, {
defaultPattern: 'phi', // Mathematical pattern to use (phi, pi, sqrt2, sqrt3, e)
adaptiveMode: true, // Automatically select optimal patterns
recursionDepth: 3, // Maximum recursion depth for compression
compressThreshold: 1024, // Min size in bytes to trigger compression
cacheResults: true // Cache compression results
});
Configuration Options
The Merlin client can be configured with various options:
Option | Type | Default | Description |
---|---|---|---|
defaultPattern | string | 'phi' | Default mathematical pattern for compression. |
adaptiveMode | boolean | true | Automatically select optimal compression patterns based on data. |
recursionDepth | number | 3 | Maximum recursion depth for compression (1-7). |
compressThreshold | number | 1024 | Minimum size in bytes to trigger compression. |
cacheResults | boolean | true | Cache compression results for improved performance. |
workerCount | number | 2 | Number of parallel workers for compression. |
asyncCompression | boolean | true | Use asynchronous compression when possible. |
hashAlgorithm | string | 'phi-hash' | Hash algorithm for data checksums. |
Compression
The Merlin API provides methods for compressing data using harmonic patterns.
Basic Compression
compress()
Compresses data using the configured harmonic patterns.
Syntax
async compress(data: Buffer | ArrayBuffer | Uint8Array | string, options?: CompressionOptions): Promise
Parameters
Name | Type | Description |
---|---|---|
data | Buffer | ArrayBuffer | Uint8Array | string | The data to compress. Strings will be encoded using UTF-8. |
options | CompressionOptions | Optional. Additional options for the compression operation. |
Returns
A Promise that resolves to a CompressionResult object containing the compressed data and metadata.
Example
// Compress a string
const textData = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit...';
const result = await merlin.compress(textData);
console.log(`Original size: ${result.originalSize} bytes`);
console.log(`Compressed size: ${result.compressedSize} bytes`);
console.log(`Compression ratio: ${result.ratio.toFixed(2)}x`);
console.log(`Pattern used: ${result.patternUsed}`);
// Compress binary data with options
const binaryData = new Uint8Array([/* ... */]);
const result2 = await merlin.compress(binaryData, {
pattern: 'pi', // Override default pattern
quality: 'max', // Maximum compression quality
lossless: true, // Ensure lossless compression
priority: 'ratio', // Prioritize compression ratio over speed
recursionDepth: 5, // Higher recursion depth
metadata: { // Custom metadata
contentType: 'image/png',
fileName: 'example.png'
}
});
Compression Options
Configure compression behavior with various options:
Option | Type | Description |
---|---|---|
pattern | string | Mathematical pattern to use ('phi', 'pi', 'sqrt2', 'sqrt3', 'e', 'auto'). |
quality | string | Compression quality ('low', 'medium', 'high', 'max'). Affects speed vs. ratio. |
lossless | boolean | Whether to use lossless compression. Default is true. |
priority | string | Optimization priority ('speed', 'ratio', 'balanced'). |
recursionDepth | number | Override the default recursion depth (1-7). |
metadata | object | Custom metadata to include with the compressed data. |
preAnalyze | boolean | Whether to analyze data before compression to optimize parameters. |
segmentSize | number | Size of data segments for parallel processing (in bytes). |
Batch Compression
compressBatch()
Compresses multiple data items in a single operation.
Syntax
async compressBatch(items: Array<{data: Buffer | ArrayBuffer | Uint8Array | string, options?: CompressionOptions}>, batchOptions?: BatchOptions): Promise>
Returns
A Promise that resolves to an array of CompressionResult objects, one for each input item.
Example
// Compress multiple items
const results = await merlin.compressBatch([
{
data: textData1,
options: { pattern: 'phi', quality: 'high' }
},
{
data: imageData,
options: { pattern: 'pi', lossless: true }
},
{
data: jsonData,
options: { priority: 'ratio' }
}
], {
parallel: true, // Process in parallel
progressCallback: (progress) => {
console.log(`Compression progress: ${progress.toFixed(2)}%`);
}
});
// Log results
results.forEach((result, index) => {
console.log(`Item ${index + 1}:`);
console.log(` Original size: ${result.originalSize} bytes`);
console.log(` Compressed size: ${result.compressedSize} bytes`);
console.log(` Ratio: ${result.ratio.toFixed(2)}x`);
});
Decompression
The Merlin API provides methods for decompressing previously compressed data.
Basic Decompression
decompress()
Decompresses data that was previously compressed with Merlin.
Syntax
async decompress(compressedData: Buffer | ArrayBuffer | Uint8Array, options?: DecompressionOptions): Promise
Parameters
Name | Type | Description |
---|---|---|
compressedData | Buffer | ArrayBuffer | Uint8Array | The compressed data to decompress. |
options | DecompressionOptions | Optional. Additional options for the decompression operation. |
Returns
A Promise that resolves to a DecompressionResult object containing the decompressed data and metadata.
Example
// Decompress data
const compressedData = result.data; // From previous compression
const decompressed = await merlin.decompress(compressedData);
console.log(`Decompressed size: ${decompressed.data.byteLength} bytes`);
console.log(`Original checksum verified: ${decompressed.checksumVerified}`);
// Get decompressed data as string
const textContent = decompressed.toString('utf8');
// Decompress with options
const decompressed2 = await merlin.decompress(compressedData, {
outputFormat: 'string', // Get result as string
encoding: 'utf8', // Output encoding
verifyChecksum: true, // Verify data integrity
metadata: true // Include original metadata
});
Decompression Options
Configure decompression behavior with various options:
Option | Type | Description |
---|---|---|
outputFormat | string | Output format ('buffer', 'arrayBuffer', 'uint8Array', 'string'). |
encoding | string | Character encoding when outputFormat is 'string' (e.g., 'utf8', 'ascii'). |
verifyChecksum | boolean | Whether to verify data integrity using the embedded checksum. |
metadata | boolean | Whether to extract and include the original metadata. |
forceLossless | boolean | Force lossless decompression mode if possible. |
Batch Decompression
decompressBatch()
Decompresses multiple compressed data items in a single operation.
Syntax
async decompressBatch(items: Array<{data: Buffer | ArrayBuffer | Uint8Array, options?: DecompressionOptions}>, batchOptions?: BatchOptions): Promise>
Returns
A Promise that resolves to an array of DecompressionResult objects, one for each input item.
Example
// Decompress multiple items
const decompressedResults = await merlin.decompressBatch([
{ data: compressedItem1 },
{ data: compressedItem2, options: { outputFormat: 'string' } },
{ data: compressedItem3 }
], {
parallel: true,
progressCallback: (progress) => {
console.log(`Decompression progress: ${progress.toFixed(2)}%`);
}
});
Harmonic Patterns
Merlin uses mathematical patterns derived from natural constants to optimize compression. Different patterns are suited to different types of data.
Available Patterns
Pattern | Mathematical Basis | Optimal Data Types |
---|---|---|
phi | Golden Ratio (φ = 1.618...) | Natural language text, organic imagery, recursive structures |
pi | Pi (π = 3.14159...) | Scientific data, circular/cyclical patterns, audio waveforms |
sqrt2 | Square Root of 2 (√2 = 1.414...) | Binary data, computer-generated content, geometric data |
sqrt3 | Square Root of 3 (√3 = 1.732...) | 3D coordinates, triangular meshes, crystal structures |
e | Euler's Number (e = 2.718...) | Growth patterns, logarithmic data, financial information |
auto | Adaptive selection | Mixed content, unknown data types |
Pattern Analysis
analyzePattern()
Analyzes data to determine the optimal compression pattern.
Syntax
async analyzePattern(data: Buffer | ArrayBuffer | Uint8Array | string, options?: AnalysisOptions): Promise
Returns
A Promise that resolves to a PatternAnalysisResult object containing the recommended patterns and analysis details.
Example
// Analyze data to find optimal pattern
const analysis = await merlin.analyzePattern(data, {
detailed: true, // Include detailed breakdown
testCompression: true // Run test compressions
});
console.log('Recommended pattern:', analysis.recommendedPattern);
console.log('Pattern scores:');
for (const [pattern, score] of Object.entries(analysis.patternScores)) {
console.log(` ${pattern}: ${score.toFixed(2)}`);
}
// Use recommended pattern for compression
const result = await merlin.compress(data, {
pattern: analysis.recommendedPattern
});
Understanding Pattern Selection
The effectiveness of each pattern depends on the inherent mathematical structures within your data. When in doubt, use the 'auto' setting to let Merlin analyze and select the optimal pattern, or run the analyzePattern() method for a detailed analysis.
Custom Patterns
Advanced users can define custom compression patterns:
// Define a custom harmonic pattern
const customPattern = {
name: 'fibonacci',
baseValues: [1, 1, 2, 3, 5, 8, 13, 21, 34, 55],
transformations: [
{ type: 'normalize', factor: 0.1 },
{ type: 'rotate', angle: Math.PI / 4 }
],
recursiveFunction: (n) => n <= 1 ? n : customPattern.recursiveFunction(n-1) + customPattern.recursiveFunction(n-2)
};
// Register custom pattern
await merlin.registerPattern(customPattern);
// Use custom pattern
const result = await merlin.compress(data, {
pattern: 'fibonacci'
});
Streaming API
For large data sets or continuous data streams, Merlin provides a streaming API that processes data in chunks.
Compression Stream
createCompressionStream()
Creates a transform stream for compressing data.
Syntax
createCompressionStream(options?: CompressionOptions): MerlinCompressionStream
Returns
A MerlinCompressionStream instance that can be used in a standard Node.js pipeline.
Example
import { pipeline } from 'stream/promises';
import fs from 'fs';
// Create compression stream
const compressionStream = merlin.createCompressionStream({
pattern: 'phi',
quality: 'high',
chunkSize: 65536 // 64 KB chunks
});
// Set up event listeners
compressionStream.on('progress', (progress) => {
console.log(`Compression progress: ${progress.toFixed(2)}%`);
});
compressionStream.on('stats', (stats) => {
console.log(`Current ratio: ${stats.currentRatio.toFixed(2)}x`);
console.log(`Bytes processed: ${stats.bytesProcessed}`);
});
// Set up a pipeline
await pipeline(
fs.createReadStream('largefile.dat'),
compressionStream,
fs.createWriteStream('largefile.dat.mln')
);
console.log('Compression complete');
console.log('Final stats:', compressionStream.getFinalStats());
Decompression Stream
createDecompressionStream()
Creates a transform stream for decompressing data.
Syntax
createDecompressionStream(options?: DecompressionOptions): MerlinDecompressionStream
Returns
A MerlinDecompressionStream instance that can be used in a standard Node.js pipeline.
Example
import { pipeline } from 'stream/promises';
import fs from 'fs';
// Create decompression stream
const decompressionStream = merlin.createDecompressionStream({
verifyChecksum: true
});
// Set up a pipeline
await pipeline(
fs.createReadStream('largefile.dat.mln'),
decompressionStream,
fs.createWriteStream('largefile_restored.dat')
);
console.log('Decompression complete');
Web Streaming API
For browser environments, Merlin supports the Web Streams API:
// Browser environment using Web Streams API
// Compress a stream from fetch
const response = await fetch('https://example.com/largefile.dat');
const readableStream = response.body;
// Create compression transformer
const compressionTransform = merlin.createCompressionTransformer({
pattern: 'sqrt2',
quality: 'high'
});
// Create transformed stream
const compressedStream = readableStream.pipeThrough(compressionTransform);
// Use the compressed stream
// e.g., save to a file using StreamSaver.js
const fileStream = streamSaver.createWriteStream('compressed.mln');
compressedStream.pipeTo(fileStream);
Optimization
Merlin provides methods to optimize compression for specific use cases and data types.
Compression Profiles
createProfile()
Creates a reusable compression profile optimized for specific data types.
Syntax
async createProfile(name: string, sampleData: Array, options?: ProfileOptions): Promise
Returns
A Promise that resolves to a CompressionProfile object that can be used for future compressions.
Example
// Create a profile optimized for JSON data
const jsonSamples = [
JSON.stringify({ id: 1, name: 'Test', data: [...] }),
JSON.stringify({ id: 2, name: 'Sample', data: [...] }),
// More samples...
];
const jsonProfile = await merlin.createProfile('json-data', jsonSamples, {
iterationCount: 5, // Number of optimization iterations
optimizeFor: 'ratio', // Optimize for maximum compression
persistent: true // Save profile for future sessions
});
console.log('Profile created:', jsonProfile.name);
console.log('Estimated compression ratio:', jsonProfile.estimatedRatio);
// Use the profile for compression
const compressedJson = await merlin.compress(jsonData, {
profile: 'json-data' // Reference profile by name
});
Managing Profiles
// List available profiles
const profiles = await merlin.listProfiles();
console.log('Available profiles:', profiles);
// Get profile details
const profile = await merlin.getProfile('json-data');
console.log('Profile details:', profile);
// Update an existing profile with new samples
await merlin.updateProfile('json-data', newSamples, {
merge: true // Merge with existing optimization data
});
// Delete a profile
await merlin.deleteProfile('json-data');
Performance Tuning
optimizePerformance()
Optimizes Merlin's performance for the current environment.
Syntax
async optimizePerformance(options?: PerformanceOptions): Promise
Returns
A Promise that resolves to a PerformanceResult object with recommended settings.
Example
// Optimize Merlin for the current environment
const performance = await merlin.optimizePerformance({
testDuration: 5000, // Run tests for 5 seconds
balanceFactors: {
speed: 0.7, // Prioritize speed (70%)
memory: 0.3 // Consider memory usage (30%)
}
});
console.log('Optimized settings:', performance.recommendedSettings);
// Apply recommended settings
merlin.applySettings(performance.recommendedSettings);
Error Handling
The Merlin API provides structured error handling for managing different types of failures.
Error Types
Merlin defines several error types for specific scenarios:
Error Class | Description | Common Causes |
---|---|---|
MerlinError | Base class for all Merlin errors | General Merlin-related issues |
CompressionError | Error during compression | Invalid data format, resource limitations |
DecompressionError | Error during decompression | Corrupted data, incompatible format |
ChecksumError | Data integrity verification failed | Data corruption, transmission errors |
PatternError | Error with compression pattern | Invalid pattern definition, incompatible data |
ResourceError | Insufficient resources | Memory limitations, CPU constraints |
Error Handling Examples
// Try-catch error handling
try {
const compressed = await merlin.compress(data);
// Use compressed data...
} catch (error) {
if (error instanceof CompressionError) {
console.error('Compression failed:', error.message);
console.error('Details:', error.details);
} else if (error instanceof ResourceError) {
console.error('Resource limitation:', error.message);
console.error('Recommended max size:', error.recommendedMaxSize);
} else {
console.error('Unexpected error:', error);
}
}
// Handling decompression errors
try {
const decompressed = await merlin.decompress(compressedData);
// Use decompressed data...
} catch (error) {
if (error instanceof ChecksumError) {
console.error('Data integrity check failed:', error.message);
// Attempt recovery
try {
const recovered = await merlin.decompress(compressedData, {
verifyChecksum: false,
attemptRecovery: true
});
console.log('Data recovered with potential loss');
} catch (recoveryError) {
console.error('Recovery failed:', recoveryError.message);
}
} else {
console.error('Decompression error:', error);
}
}
Error Recovery
Some Merlin errors provide recovery methods:
try {
const decompressed = await merlin.decompress(compressedData);
} catch (error) {
if (error.canRecover()) {
// Get recovery options
const recoveryOptions = error.getRecoveryOptions();
console.log('Recovery options available:', recoveryOptions);
// Attempt recovery with recommended options
try {
const recovered = await error.recover();
console.log('Recovery successful. Recovered data:', recovered);
} catch (recoveryError) {
console.error('Recovery failed:', recoveryError);
}
} else {
console.error('Unrecoverable error:', error);
}
}