Data & Analytics

54 of 54 repositories

• firecrawl/firecrawl

  Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

  AI summary: An API service that crawls websites and turns them into clean, LLM-ready markdown data.

  188,550dataTypeScriptAGPL-3.0aiai-agentsai-crawlerai-scraping

• microsoft/markitdown

  Python tool for converting files and office documents to Markdown.

  AI summary: A lightweight Python utility for converting various file formats into clean Markdown.

  188,396dataPythonMITautogenautogen-extensionlangchainmarkdown

• TauricResearch/TradingAgents

  TradingAgents: Multi-Agents LLM Financial Trading Framework

  AI summary: A robust research framework for developing, testing, and deploying autonomous AI agents for algorithmic financial trading.

  109,751dataPythonApache-2.0agentfinancellmmultiagent

• koala73/worldmonitor

  Real-time global intelligence dashboard. AI-powered news aggregation, geopolitical monitoring, and infrastructure tracking in a unified situational awareness interface

  AI summary: A real-time global intelligence dashboard utilizing AI and dual map engines to track geopolitical and infrastructure signals.

  87,764dataTypeScriptAGPL-3.0agentaidashboardgeopolitics

• opendatalab/MinerU

  Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

  AI summary: High-accuracy parsing engine that converts complex documents into LLM-ready Markdown and JSON.

  81,001dataPythonOtherai4sciencedocument-analysisdocxextract-data

• elastic/elasticsearch

  Free and Open Source, Distributed, RESTful Search Engine

  AI summary: Distributed search, analytics engine, and vector database optimized for speed and relevance.

  78,188dataJavaOtherelasticsearchjavasearch-engineanalytics

• openbq-org/OpenBB

  Open Data Platform for analysts, quants and AI agents.

  AI summary: An open-source investment research software platform offering extensive financial data access and analytics.

  73,764dataPythonOtheraicryptoderivativeseconomics

• docling-project/docling

  Get your documents ready for gen AI

  AI summary: A document processing tool that parses diverse formats like PDFs into machine-readable data for generative AI.

  68,375dataPythonMITaiconvertdocument-parserdocument-parsing

• NanmiCoder/MediaCrawler

  小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫

  AI summary: An automated data mining script for scraping user profiles and posts from major Chinese social media platforms.

  66,143dataPythonOtherscraperplaywrightdata-miningpython

• ZhuLinsen/daily_stock_analysis

  LLM 驱动的多市场股票智能分析系统：多源行情、实时新闻、决策看板与自动推送，支持零成本定时运行。 LLM-powered multi-market stock analysis system with multi-source market data, real-time news, decision dashboard, automated notifications, and cost-free scheduled runs.

  AI summary: An automated system for scraping, analyzing, and visualizing daily A-share stock market data.

  65,883dataPythonMITa-stockai-agentaigcllm

• virattt/ai-hedge-fund

  An AI Hedge Fund Team

  AI summary: An experimental framework for simulating algorithmic trading strategies using multi-agent LLM consensus.

  63,840dataPythonMITalgorithmic-tradingfinancemulti-agentllm

• sansan0/TrendRadar

  ⭐AI-driven public opinion & trend monitor with multi-platform aggregation, RSS, and smart alerts.🎯 告别信息过载，你的 AI 舆情监控助手与热点筛选工具！聚合多平台热点 + RSS 订阅，支持关键词精准筛选。AI 智能筛选新闻 + AI 翻译 + AI 分析简报直推手机，也支持接入 MCP 架构，赋能 AI 自然语言对话分析、情感洞察与趋势预测等。支持 Docker ，数据本地/云端自持。集成微信/飞书/钉钉/Telegram/邮件/ntfy/bark/slack 等渠道智能推送。

  AI summary: An automated intelligence gathering tool that monitors and summarizes emerging tech trends across platforms.

  62,648dataPythonGPL-3.0aibarkdata-analysisdocker

• KeygraphHQ/shannon

  Shannon is an AI pentester for web applications and APIs. It analyzes your source code, identifies attack vectors, and executes real exploits to prove vulnerabilities before they reach production.

  AI summary: A high-performance, open-source knowledge graph engine optimized for fast querying and AI integration.

  48,554dataTypeScriptAGPL-3.0ai-penetration-testingai-securityapi-securityappsec

• bilawalsidhu/gods-eye-view

  A spy satellite simulator in your browser, except the data is real. Live open source spatial intelligence on a photorealistic 3D globe.

  AI summary: Browser-based spy satellite simulator offering live open-source spatial intelligence on a photorealistic 3D globe.

  47,427dataJavaScriptOther3d-globecesiumflight-trackinggeospatial

• PostHog/posthog

  :hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

  AI summary: A comprehensive, open-source product analytics suite providing tools for tracking, feature flags, and session recording.

  40,123dataPythonOtherab-testingai-analyticsanalyticscdp

• google/langextract

  A Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.

  AI summary: AI-powered tool for extracting structured data from unstructured text using large language models.

  38,927dataPythonApache-2.0geminigemini-aigemini-apigemini-flash

• Fincept-Corporation/FinceptTerminal

  FinceptTerminal is a modern finance application offering advanced market analytics, investment research, and economic data tools, designed for interactive exploration and data-driven decision-making in a user-friendly environment.

  AI summary: A native C++ financial intelligence platform combining institutional analytics with local AI agents.

  32,160dataC++Otherai-agentsalgorithmic-tradingbloomberg-terminalcpp

• topoteretes/cognee

  Cognee is the open-source AI memory platform for agents. Give your AI agents persistent long-term memory with small models for free

  AI summary: An open-source AI memory platform that provides persistent long-term memory and knowledge graphs for agents.

  31,326dataPythonApache-2.0agent-memoryagent-skillsaiai-agents

• supermemoryai/supermemory

  Memory and context engine + app that is extremely fast, scalable, and can be run fully locally. The Memory API for the AI era.

  AI summary: A scalable, local-first memory and context engine designed to serve as a personal or company brain for AI applications.

  31,067dataTypeScriptMITagent-memoryai-memorycloudflare-kvcloudflare-pages

• nautechsystems/nautilus_trader

  Production-grade Rust-native trading engine with deterministic event-driven architecture

  AI summary: A highly deterministic, Rust-native algorithmic trading engine for multiple asset classes.

  29,562dataRustLGPL-3.0algorithmic-trading-engineartificial-intelligencecrypto-tradingequity-trading

• opendataloader-project/opendataloader-pdf

  PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.

  AI summary: A high-accuracy PDF parser that extracts AI-ready Markdown, JSON, and HTML using deterministic and hybrid approaches.

  29,452dataJavaApache-2.0a11yaccessibilityaibounding-box

• TencentCloud/TencentDB-Agent-Memory

  TencentDB Agent Memory is a team-level memory hub for AI Agents — turning conversations, docs, and code into four reusable memory assets (Chat Memory, Skill, LLM-Wiki, Code-Graph) that are governed, shared, and equipped across agents and frameworks.

  AI summary: A team-level memory hub governing chat, skills, wikis, and code graphs for AI agents.

  27,682dataTypeScriptOtheragentai-agentembeddingllm

• tursodatabase/turso

  A SQL database in Rust: SQLite-compatible, now also speaking Postgres (experimental). The LLVM of databases.

  AI summary: An edge-hosted, globally distributed SQL database built on libSQL, compatible with SQLite.

  24,527dataRustMITdatabaseembedded-databasesqlsqlite3

• hasaneyldrm/exercises-dataset

  1,324-exercise fitness dataset — animation GIFs, 180×180 thumbnails, muscle-group & equipment data, and step-by-step instructions in 6 languages. The exercise data layer behind the LogPress app.

  AI summary: A comprehensive JSON dataset detailing thousands of fitness exercises with categorical metadata.

  22,463dataHTMLOtherdatasetexercise-databaseexercisesfitness

• firecrawl/pdf-inspector

  Fast Rust library for PDF inspection, classification, and text extraction. Intelligently detects scanned vs text-based PDFs to enable smart routing decisions.

  AI summary: A high-performance Rust library for intelligent PDF classification, layout-aware text extraction, and clean Markdown conversion.

  19,462dataRustMITmarkdownnodejsocr-routingpdf

• microsoft/data-formulator

  🪄 Data Formulator is an interactive AI-powered data analysis system makes it easy to connect, explore and visualize data.

  AI summary: An interactive, AI-powered data analysis system that generates visualizations from raw datasets automatically.

  17,514dataPythonMITdata-visualizationai-agentspythonanalytics

• RyanCodrai/turbovec

  A vector index built on TurboQuant, written in Rust with Python bindings

  AI summary: A memory-efficient Rust vector index with Python bindings, built on Google's TurboQuant algorithm.

  17,277dataRustMITannavx512embeddingembeddings

• xbtlin/ai-berkshire

  AI 时代的伯克希尔：基于 Claude Code / Codex 的价值投资研究框架。巴菲特·芒格·段永平·李录四大师方法论 + 多Agent并行研究。| AI-era Berkshire: a value investing research framework built for Claude Code / Codex. 4 masters' methodologies + multi-agent adversarial analysis.

  AI summary: An institutional-grade, multi-agent investment research framework combining fundamental analysis with LLM automation.

  16,603dataHTMLMITaiai-agentanthropicberkshire-hathaway

• alibaba/zvec

  A lightweight, lightning-fast, in-process vector database

  AI summary: An open-source, in-process vector database by Alibaba built for lightweight, low-latency similarity search.

  16,063dataC++Apache-2.0agent-skillsdbembeddedfaiss

• run-llama/liteparse

  A fast, helpful, and open-source document parser

  AI summary: A lightweight parser for extracting and structuring text from complex document formats.

  12,769dataRustApache-2.0document-ocrdocument-processingocrocr-recognition

• datalab-to/chandra

  OCR model that handles complex tables, forms, handwriting with full layout.

  AI summary: A state-of-the-art document intelligence library for extracting structured data from complex documents using AI.

  12,416dataPythonApache-2.0aiocrdocument-intelligencedata-extraction

• apache/cassandra

  Open source transactional distributed database. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure without compromising performance.

  AI summary: Highly scalable, partitioned row store database designed for zero single points of failure.

  10,113dataJavaApache-2.0cassandradatabasejavanosql

• xberg-io/xberg

  Polyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. Fifteen bindings, with CLI, REST API, and MCP server.

  AI summary: A high-performance document intelligence framework for extracting structured data from over 98 file formats.

  9,366dataRustMITbuncsharpdocument-intelligenceelixir

• GoogleCloudPlatform/knowledge-catalog

  Google Cloud Knowledge Catalog Tools and Samples

  AI summary: An AI-powered data catalog providing a dynamic knowledge graph for context-aware enterprise AI agents.

  9,350dataTypeScriptApache-2.0knowledge-graphdata-catalogmetadataai-context

• google-deepmind/weathernext

  AI summary: A highly accurate global, medium-range atmospheric forecasting model developed by Google DeepMind.

  7,694dataPythonApache-2.0weatherweather-forecastmachine-learningclimate

• wpzzz/blocked-sites-in-south-korea

  AI summary: A maintained dataset and analysis of websites blocked by South Korean internet service providers.

  7,510dataPythoncensorshipnetwork-analysissouth-koreadataset

• The-Swarm-Corporation/AutoHedge

  Build your autonomous hedge fund in minutes. AutoHedge harnesses the power of swarm intelligence and AI agents to automate market analysis, risk management, and trade execution.

  AI summary: An enterprise-grade autonomous hedge fund system powered by a multi-agent pipeline and swarm intelligence.

  6,232dataPythonMITaiblackrockfinancegoldmansachs

• QuipNetwork/xquad

  A rust implementation of the Quip Network's quantum virtual machine.

  AI summary: A hardware-agnostic Rust toolchain for expressing and running quadratic optimization models across quantum and classical solvers.

  5,585dataPythonAGPL-3.0quantum-computingoptimizationrustvirtual-machine

• shy3130/tick-stock-panel

  TSP自托管、零运维的 A 股「选股 + 监控 + 回测」量化工作台 | LLM能力驱使策略定制+个股分析+复盘 | 自由接入第三方数据源与个性化扩展数据 | 个人开源

  AI summary: A self-hosted, zero-maintenance quantitative workspace for screening, monitoring, and backtesting Chinese A-shares.

  5,462dataPythonMITa-stockai-agentaigcaigc-detection

• malisper/pgrust

  Postgres rewritten in Rust, now faster than Postgres and Clickhouse

  AI summary: A high-performance Rust reimplementation of PostgreSQL 18 designed to be 300x faster for analytics.

  5,220dataRustAGPL-3.0ai-assisted-developmentdatabasepostgrespostgresql