⚡ Built for Performance¶
Hazy runs hashing and sketch updates in a Rust core, with batch paths that release the Python GIL.
What is Hazy?¶
Hazy provides probabilistic data structures that trade perfect accuracy for massive space and time savings. These structures are essential for big data applications where exact answers are impractical.
from hazy import BloomFilter, HyperLogLog, CountMinSketch
# Check if items exist (with 1% false positive rate)
users = BloomFilter(expected_items=1_000_000, false_positive_rate=0.01)
users.add("alice@example.com")
print("alice@example.com" in users) # True
# Count unique items using only 16KB of memory
counter = HyperLogLog(precision=14)
for user_id in million_user_ids:
counter.add(user_id)
print(f"Unique users: {counter.cardinality():,.0f}") # ~1,000,000
# Track frequencies in streaming data
clicks = CountMinSketch(width=10000, depth=5)
clicks.add("/home")
print(f"Home page clicks: {clicks['/home']}")
Data Structures¶
-
Bloom Filter
Space-efficient set membership testing. Know if an item is definitely not in a set, or probably in it.
-
HyperLogLog
Count unique items with ~1% error using just 16KB. Perfect for cardinality estimation at scale.
-
Count-Min Sketch
Estimate frequencies in streaming data. Track how often items appear without storing them all.
-
MinHash
Estimate set similarity using Jaccard index. Find similar documents, users, or items efficiently.
-
Cuckoo Filter
A fixed-capacity membership filter with deletion support.
-
Top-K
Find the most frequent items in a stream using the Space-Saving algorithm with bounded memory.
Why Hazy?¶
Rust Performance¶
Implemented in Rust with PyO3 bindings and explicit batch operations. Uses xxHash3 for deterministic, seeded hashing.
Simple API¶
Pythonic interface with in operator, len(), serialization, and file I/O. Feels natural and intuitive.
Compact Serialization¶
Byte and JSON serialization, plus file helpers for saving and restoring structures.
Visualization¶
Built-in plotting with matplotlib for debugging and understanding your data structures.
Quick Comparison¶
| Structure | Use Case | Memory | Error |
|---|---|---|---|
| BloomFilter | Set membership | ~1.2 bytes/item | Configurable FPR |
| HyperLogLog | Cardinality | 2^p bytes | ~1.04/√(2^p) |
| CountMinSketch | Frequencies | w × d × 8 bytes | ε·N overestimate |
| MinHash | Similarity | 8 × k bytes | 1/√k |
| CuckooFilter | Membership + delete | ~1 byte/item | ~3% FPR |
| TopK | Heavy hitters | O(k) | Bounded |
Installation¶
Featured Examples¶
-
Web Analytics
Track unique visitors, page views, and trending content with minimal memory.
-
Deduplication
Detect duplicate events, URLs, or records in streaming data pipelines.
-
Similarity Search
Find similar documents or detect near-duplicates using MinHash signatures.