OSDU® Model Context Protocol (MCP) Server & HTTP REST API

A Python implementation that provides two ways to interact with OSDU® (Open Subsurface Data Universe) platform APIs:

1. MCP Server: Model Context Protocol server for AI agents (Claude Desktop, custom AgenticAI)
2. HTTP REST API: Standard REST API for web applications and services

Both interfaces provide the same comprehensive tools for search, storage, schema management, and dataset operations on OSDU® data.

OSDU® and the OSDU logo are registered trademarks of The Open Group.

Features

Core Functionality

• OSDU® Search Service: Query and search records using structured queries
• OSDU® Storage Service: Complete CRUD operations for records and versions
• OSDU® Schema Service: Dynamic schema discovery and validation
• OSDU® Dataset Service: Dataset creation, management, and file operations
• OpenAPI Compliance: All tools generated from official OSDU® OpenAPI specifications
• Authentication: JWT token-based authentication with OSDU®
• Error Handling: Comprehensive error handling and logging
• Configuration: Environment-based configuration management

Dual Interface Support

• MCP Protocol: JSON-RPC 2.0 over stdin/stdout for AI agents
• HTTP REST API: Standard REST endpoints with FastAPI and automatic OpenAPI documentation
• Same Tools: Identical functionality available through both interfaces
• Flexible Deployment: Run as MCP server, HTTP API, or both simultaneously

Installation

1. Clone the repository:

git clone <repository-url>
cd osdu-mcp

1. Install dependencies:

pip install -e .

1. Configure environment variables:

cp .env.example .env
# Edit .env with your OSDU configuration

OSDU Subscription

If you are looking for a ready to use OSDU instance, simply subscribe to Express for EDI in AWS Marketplace.

Configuration

Set the following environment variables in your .env file:

• OSDU_BASE_URL: Base URL of your OSDU® platform instance
• OSDU_DATA_PARTITION_ID: Data partition identifier
• OSDU_AUTHORIZATION: JWT authorization token (with "Bearer " prefix)

Usage

Running the MCP Server

#### Prerequisites

Before starting the server, ensure:

1. Environment Configuration: Set up your .env file with OSDU® credentials
2. Network Access: Ensure connectivity to your OSDU® platform instance
3. Authentication: Valid JWT token with appropriate permissions
4. Dependencies: Install project dependencies with pip install -e .

#### Starting the Server

The OSDU MCP server communicates via stdin/stdout using the JSON-RPC 2.0 protocol. Choose one of these methods to start:

Method 1: Command Line Entry Point (Recommended)

# From project root directory
osdu-mcp

Method 2: Python Module Execution

# From project root directory
python -m osdu_mcp.server

Method 3: Direct Script Execution

# From project root directory
python osdu_mcp/server.py

Method 4: With Virtual Environment

# Activate virtual environment first
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Then use any of the above methods
osdu-mcp
# OR
python -m osdu_mcp.server

#### Server Behavior

When started, the MCP server:

• Loads configuration from .env file or environment variables
• Initializes all OSDU service tools (29 tools across 4 services)
• Listens for MCP protocol messages on stdin
• Responds with JSON-RPC 2.0 formatted messages on stdout
• Runs continuously until terminated

Expected startup behavior:

• Server starts silently (no console output unless errors occur)
• Waits for MCP client connections via stdin/stdout
• Ready to receive list_tools and call_tool requests

#### Stopping the Server

Interactive Mode:

# Press Ctrl+C to stop the server
^C

Background Process:

# If running in background, find and kill the process
ps aux | grep osdu-mcp
kill <process_id>

# Or use pkill
pkill -f "osdu-mcp"
pkill -f "osdu_mcp.server"

Docker/Container Environment:

# Send SIGTERM signal
docker stop <container_name>

# Or force stop
docker kill <container_name>

#### Service Management

Running as a Service (Linux/macOS)

Create a systemd service file at /etc/systemd/system/osdu-mcp.service:

[Unit]
Description=OSDU MCP Server
After=network.target

[Service]
Type=simple
User=your-user
WorkingDirectory=/path/to/osdu-mcp
Environment=PATH=/path/to/osdu-mcp/venv/bin
ExecStart=/path/to/osdu-mcp/venv/bin/osdu-mcp
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Manage the service:

# Start service
sudo systemctl start osdu-mcp

# Stop service  
sudo systemctl stop osdu-mcp

# Enable auto-start on boot
sudo systemctl enable osdu-mcp

# Check status
sudo systemctl status osdu-mcp

# View logs
journalctl -u osdu-mcp -f

Process Monitoring

# Check if server is running
ps aux | grep osdu-mcp

# Monitor server resources
top -p $(pgrep -f osdu-mcp)

# View server logs (if logging to file)
tail -f /path/to/logfile.log

Integration with MCP Clients

This server integrates with MCP-compatible clients through stdio communication:

#### Claude Desktop Integration

Add to your Claude Desktop configuration (claude_desktop_config.json):

{
  "mcpServers": {
    "osdu-mcp": {
      "command": "osdu-mcp",
      "args": [],
      "env": {
        "OSDU_BASE_URL": "https://your-osdu-instance.com",
        "OSDU_DATA_PARTITION_ID": "your-partition-id", 
        "OSDU_AUTHORIZATION": "Bearer your-jwt-token"
      }
    }
  }
}

or directly map to the code

{
  "mcpServers": {
    "osdu-mcp": {
      "command": "/path/tp/osdu-mcp/venv/bin/python",
      "args": ["-m", "osdu_mcp.server"],
      "env": {
        "OSDU_BASE_URL": "https://your-osdu-instance.com",
        "OSDU_DATA_PARTITION_ID": "your-partition-id", 
        "OSDU_AUTHORIZATION": "Bearer your-jwt-token"
      }
    }
  }
}

#### Custom MCP Client Integration

The server follows the standard MCP protocol:

import subprocess
import json

# Start the MCP server
process = subprocess.Popen(
    ['osdu-mcp'],
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    text=True
)

# Send MCP requests
request = {
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/list"
}

process.stdin.write(json.dumps(request) + '\n')
process.stdin.flush()

# Read response
response = process.stdout.readline()
tools = json.loads(response)

#### Available Clients

• Claude Desktop: AI assistant with MCP support
• Custom Clients: Build using MCP SDK libraries
• AI Development Environments: Any platform supporting MCP protocol
• Chat Interfaces: Integration with various AI chat applications

#### Connection Testing

# Test basic connectivity
echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | osdu-mcp

For detailed MCP protocol documentation, visit: https://modelcontextprotocol.io

HTTP REST API Server

Starting the HTTP API Server

The HTTP REST API provides the same OSDU® tools through standard REST endpoints with automatic OpenAPI documentation.

Method 1: Command Line Entry Point (Recommended)

# Start HTTP server on default port 8000
osdu-http

Method 2: Python Module Execution

# Start with custom configuration
python -m osdu_mcp.http_server

Method 3: Using uvicorn directly

# Start with custom host/port
uvicorn osdu_mcp.http_server:create_app --host 0.0.0.0 --port 8080

Method 4: With Docker

# Build and run with Docker
docker build -t osdu-api .
docker run -p 8000:8000 --env-file .env osdu-api

# Or use docker-compose
docker-compose up osdu-api

HTTP Configuration

Add these optional environment variables to your .env file:

# HTTP Server Configuration
HTTP_HOST=0.0.0.0          # Server host (default: 0.0.0.0)
HTTP_PORT=8000             # Server port (default: 8000)  
HTTP_RELOAD=false          # Auto-reload for development (default: false)

API Endpoints

The HTTP API provides the following endpoints:

#### Core Endpoints

• GET / - API information and available endpoints
• GET /health - Health check and configuration status
• GET /tools - List all available OSDU® tools with schemas
• POST /tools/call - Execute any tool by name
• GET /docs - Interactive OpenAPI documentation (Swagger UI)
• GET /redoc - Alternative API documentation (ReDoc)

#### Tool-Specific Endpoints

• POST /tools/search/{tool_name} - Execute search tools
• POST /tools/storage/{tool_name} - Execute storage tools
• POST /tools/schema/{tool_name} - Execute schema tools
• POST /tools/dataset/{tool_name} - Execute dataset tools

Usage Examples

#### List Available Tools

curl http://localhost:8000/tools

#### Execute a Search Query

curl -X POST http://localhost:8000/tools/search/query \
  -H "Content-Type: application/json" \
  -d '{
    "query": "*",
    "limit": 10,
    "kind": "*:*:*:*"
  }'

#### Create Storage Records

curl -X POST http://localhost:8000/tools/storage/create_or_update_records \
  -H "Content-Type: application/json" \
  -d '{
    "records": [
      {
        "id": "partition:well:example-001",
        "kind": "partition:wks:master-data--Well:1.0.0",
        "data": {"wellName": "Example Well"}
      }
    ]
  }'

#### Generic Tool Execution

curl -X POST http://localhost:8000/tools/call \
  -H "Content-Type: application/json" \
  -d '{
    "name": "osdu_dataset_get_registry",
    "arguments": {"dataset_id": "partition:dataset:example"}
  }'

API Documentation

Once the server is running, visit:

• Interactive Docs: http://localhost:8000/docs (Swagger UI)
• Alternative Docs: http://localhost:8000/redoc (ReDoc)
• OpenAPI Spec: http://localhost:8000/openapi.json

HTTP Response Format

All endpoints return JSON responses in this format:

{
  "success": true,
  "result": {
    // Tool-specific response data
  }
}

Error responses:

{
  "success": false,
  "error": "Error message description"
}

Production Deployment

#### Docker Deployment

# Production docker-compose
docker-compose -f docker-compose.prod.yml up -d

#### Reverse Proxy (nginx)

server {
    listen 80;
    server_name your-domain.com;
    
    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

#### Environment-Specific Configuration

# Development
HTTP_RELOAD=true LOG_LEVEL=DEBUG osdu-http

# Production  
HTTP_HOST=0.0.0.0 HTTP_PORT=8000 osdu-http

Available Tools

#### Search Tools

• osdu_search_query: Query the OSDU index with full text search, range queries, geospatial search, sorting, and aggregation support
• osdu_search_query_with_cursor: Query with cursor support for efficiently retrieving large result sets

Kind Parameter Format: The kind parameter uses OSDU string patterns:

• "*:*:*:*" - Search all record types
• "osdu:wks:well:*" - Search all well records
• "osdu:wks:well:1.0.0" - Search specific well version
• "partition:wks:master-data--Well:1.0.0" - Search specific partition and version

#### Storage Tools

• osdu_storage_create_or_update_records: Create or update records (main injection mechanism)
• osdu_storage_get_record: Get latest record version with optional field filtering
• osdu_storage_get_specific_record_version: Get specific version of a record
• osdu_storage_get_record_versions: Get all version numbers for a record
• osdu_storage_fetch_records: Fetch multiple records with field filtering
• osdu_storage_fetch_records_batch: Fetch records with normalization support
• osdu_storage_query_all_records: Get all record IDs from a kind with cursor pagination
• osdu_storage_delete_record: Logically delete a record (can be reverted)
• osdu_storage_bulk_delete_records: Soft delete multiple records
• osdu_storage_purge_record: Permanently delete record and all versions
• osdu_storage_purge_record_versions: Permanently delete specific versions
• osdu_storage_patch_records: Update records using patch operations
• osdu_storage_copy_record_references: Copy record references between namespaces

#### Schema Tools

• osdu_schema_get_schema: Get schema definition by system-defined ID
• osdu_schema_search_schemas: Search schemas with comprehensive filtering options
• osdu_schema_create_schema: Create new schema in repository (PUBLISHED status)
• osdu_schema_upsert_development_schema: Create/update schema in DEVELOPMENT status
• osdu_schema_upsert_system_schema: Create/update system schema (service principal required)

#### Dataset Tools

• osdu_dataset_register: Create or update dataset registries
• osdu_dataset_get_registry: Get dataset registry by ID
• osdu_dataset_get_registries: Get multiple dataset registries
• osdu_dataset_storage_instructions: Generate storage instructions (signed URLs)
• osdu_dataset_retrieval_instructions_single: Generate retrieval instructions for single dataset
• osdu_dataset_retrieval_instructions_multiple: Generate retrieval instructions for multiple datasets
• osdu_dataset_soft_delete: Soft delete a dataset registry
• osdu_dataset_undelete: Un-delete a soft deleted dataset
• osdu_dataset_revoke_url: Revoke previously generated URLs

Architecture

The MCP server follows a modular architecture:

• server.py: Main MCP server implementation with JSON-RPC 2.0
• client.py: HTTP client for OSDU API interactions
• config.py: Configuration management with validation
• exceptions.py: Custom exception classes
• tools/: Individual tool implementations for each OSDU service
• open_api/: OSDU-provided OpenAPI specifications used to generate tools

OpenAPI Specification Compliance

All tools are generated based on official OSDU OpenAPI specifications located in the open_api/ folder:

• `dataset.swagger.yaml`: Dataset Service API specification - used to create dataset management tools
• `search_openapi.yaml`: Search API v2 specification - used to create search and query tools
• `storage_openapi.yaml`: Storage API v2 specification - used to create record CRUD operation tools
• `schema_openapi.yaml`: Schema Service API v1 specification - used to create schema management tools

Each tool implementation strictly follows the parameters, request/response formats, and endpoints defined in these official OSDU specifications, ensuring full API compliance and compatibility.

Development

Setting Up Development Environment

1. Create and activate a virtual environment:

python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

1. Install the project in development mode:

pip install -e .

1. Install development dependencies:

pip install pytest pytest-asyncio

Running Tests

The project includes comprehensive unit and integration tests for all four OSDU® services (Search, Dataset, Schema, and Storage).

#### Run All Tests

python -m pytest tests/ -v

#### Run Specific Test Suites by Service

# Search tool tests
python -m pytest tests/test_search_tool.py tests/test_search_tool_integration.py -v

# Dataset tool tests
python -m pytest tests/test_dataset_tool.py tests/test_dataset_tool_integration.py -v

# Schema tool tests
python -m pytest tests/test_schema_tool.py tests/test_schema_tool_integration.py -v

# Storage tool tests
python -m pytest tests/test_storage_tool.py tests/test_storage_tool_integration.py -v

#### Run Individual Test Files

# Unit tests
python -m pytest tests/test_search_tool.py -v      # Search unit tests (15 tests)
python -m pytest tests/test_dataset_tool.py -v     # Dataset unit tests (21 tests) 
python -m pytest tests/test_schema_tool.py -v      # Schema unit tests (19 tests)
python -m pytest tests/test_storage_tool.py -v     # Storage unit tests (25 tests)

# Integration tests
python -m pytest tests/test_search_tool_integration.py -v    # Search integration (8 tests)
python -m pytest tests/test_dataset_tool_integration.py -v   # Dataset integration (9 tests)
python -m pytest tests/test_schema_tool_integration.py -v    # Schema integration (9 tests)  
python -m pytest tests/test_storage_tool_integration.py -v   # Storage integration (11 tests)

#### Test Structure

Search Tool Tests (23 tests):

• Unit Tests (test_search_tool.py): 15 tests covering tool schemas, parameter validation, error handling, and response formatting
• Integration Tests (test_search_tool_integration.py): 8 tests with realistic OSDU® scenarios including geospatial searches, pagination, and complex queries

Dataset Tool Tests (30 tests):

• Unit Tests (test_dataset_tool.py): 21 tests covering all 9 dataset tools, schema validation, parameter constraints, and error scenarios
• Integration Tests (test_dataset_tool_integration.py): 9 tests with complete workflows including dataset lifecycle, bulk operations, and large file handling

Schema Tool Tests (28 tests):

• Unit Tests (test_schema_tool.py): 19 tests covering all 5 schema tools, pattern validation, status/scope enums, parameter mapping, and governance workflows
• Integration Tests (test_schema_tool_integration.py): 9 tests with schema development lifecycle, system schema management, version filtering, and complex schema scenarios

Storage Tool Tests (36 tests):

• Unit Tests (test_storage_tool.py): 25 tests covering all 13 storage tools, record CRUD operations, versioning, patch operations, and bulk operations
• Integration Tests (test_storage_tool_integration.py): 11 tests with complete record lifecycle, bulk workflows, versioning, JSON Patch operations, and namespace copying

#### Expected Test Results

======================= 117 passed, 2 warnings in 0.65s ========================

Complete Test Coverage:

• Total Tests: 117 across all OSDU® services
• Unit Tests: 80 tests (15 + 21 + 19 + 25)
• Integration Tests: 37 tests (8 + 9 + 9 + 11)
• Services Covered: 4 complete OSDU® services
• Tools Covered: 29 total tools (2 Search + 9 Dataset + 5 Schema + 13 Storage)

All tests should pass. The warnings about Pydantic validators are non-critical deprecation notices.

Code Quality

#### Code Formatting

black osdu_mcp/
ruff osdu_mcp/

#### Type Checking (if mypy is installed)

mypy osdu_mcp/

Test Development

When adding new tools or modifying existing ones, follow the established testing patterns:

#### Creating Unit Tests

# Follow the naming convention: test_<service_name>_tool.py
tests/test_search_tool.py        # Search tool unit tests (15 tests)
tests/test_dataset_tool.py       # Dataset tool unit tests (21 tests)
tests/test_schema_tool.py        # Schema tool unit tests (19 tests)
tests/test_storage_tool.py       # Storage tool unit tests (25 tests)

#### Creating Integration Tests

# Follow the naming convention: test_<service_name>_tool_integration.py  
tests/test_search_tool_integration.py        # Search integration tests (8 tests)
tests/test_dataset_tool_integration.py       # Dataset integration tests (9 tests)
tests/test_schema_tool_integration.py        # Schema integration tests (9 tests)
tests/test_storage_tool_integration.py       # Storage integration tests (11 tests)

#### Test Coverage Guidelines

• Unit Tests: Test individual tool methods, schema validation, parameter handling, and error scenarios
• Integration Tests: Test realistic workflows, multi-step operations, and complex scenarios
• Mock Usage: Use AsyncMock for HTTP client operations, realistic response data
• Error Testing: Include authentication errors, API errors, and network failure scenarios

Working with OpenAPI Specifications

The project includes official OSDU OpenAPI specifications in the open_api/ directory:

open_api/
├── dataset.swagger.yaml     # Dataset Service API
├── schema_openapi.yaml      # Schema Service API v1
├── search_openapi.yaml      # Search API v2
└── storage_openapi.yaml     # Storage API v2

These specifications are the authoritative source for:

• API endpoint definitions
• Request/response schemas
• Parameter validation rules
• Authentication requirements

When extending or modifying tools, always refer to these specifications to ensure compliance.

Troubleshooting

Common Issues

#### Import Errors If you encounter import errors, ensure the package is installed in development mode:

pip install -e .

#### Test Failures If tests fail due to missing dependencies:

pip install pytest pytest-asyncio

To run specific failing tests for debugging:

# Run individual service tests
python -m pytest tests/test_search_tool.py -v --tb=long      # Search service
python -m pytest tests/test_dataset_tool.py -v --tb=long     # Dataset service
python -m pytest tests/test_schema_tool.py -v --tb=long      # Schema service  
python -m pytest tests/test_storage_tool.py -v --tb=long     # Storage service

# Run integration tests only
python -m pytest tests/*_integration.py -v --tb=long

# Run with more verbose output for all tests
python -m pytest tests/ -v -s --tb=long

#### Authentication Issues

• Verify your JWT token is valid and not expired
• Ensure the OSDU_AUTHORIZATION header includes "Bearer " prefix
• Check that your token has the required permissions for OSDU operations

#### Network Connectivity

• Verify the OSDU_BASE_URL is accessible from your environment
• Check firewall and proxy settings if applicable
• Ensure HTTPS is properly configured

#### Configuration Issues

• Verify all required environment variables are set
• Check .env file syntax and formatting
• Ensure OSDU_DATA_PARTITION_ID matches your OSDU instance

Getting Help

If you encounter issues:

1. Check the server logs for detailed error messages
2. Verify your OSDU platform connectivity outside of this tool
3. Review the OpenAPI specifications in the project for API details
4. Run tests to ensure the development environment is properly set up

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Run tests and ensure they pass
6. Submit a pull request

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.