Senior Software Engineer (Storage Cache)
Posted Updated
Description
- As a Principal Engineer on the Cache team (part of the Infra Storage org), you will innovate and operate large-scale, in-house distributed systems to solve Roblox’s ever-growing caching challenges
- You will report directly to the Engineering Manager for the Cache team
- Lead the architectural transition to a next-generation, multitenant caching service built on ValKey, ensuring strict data, resource, and failure isolation for all tenants
- Drive systemic optimizations to mitigate head-of-line blocking, manage hot keys, and maximize CPU and memory utilization across physical machine clusters
- Design and build robust frameworks to automate development, chaos testing (fault/latency injection), and monitoring for 24x7 mission-critical services, targeting 99.99%+ availability and elastic scalability
- Champion engineering best practices by leading design reviews, performance benchmarking, failure drills, and blameless post-incident retrospectives
- Mentor and empower engineers, fostering a culture of deep domain expertise and seamless knowledge sharing across the Storage, Platform, and Product teams
- Experience & Education: A BS degree in Computer Science (or equivalent professional experience) with at least 6+ years of hands-on software engineering experience
- Observability Skills: Hands-on experience with modern telemetry and observability stacks (e.g., Prometheus, Grafana, AlertManager, Kibana)
- [Bonus] Open Source Contributions: A track record of contributing to or maintaining major open-source caching projects such as Redis, ValKey, or Memcached
- [Bonus] Advanced Cache Internals: Experience extending cache functionality (e.g., writing custom Redis modules in C/Rust, complex Lua scripting) or deep-tuning underlying memory allocators like jemalloc
- Infrastructure Chops: A strong builder mindset with proven experience running Active/Active distributed systems on container orchestrators like Kubernetes or Nomad
- Programming Proficiency: Strong, hands-on programming experience in Go and C++
- Problem-Solving Track Record: Proven success in resolving massive-scale bottlenecks, such as overcoming the limitations of decentralized Gossip protocols or mitigating partial failures in distributed systems
- [Bonus] Caching Proxies & Topologies: Experience with caching proxies (e.g., Twemproxy, Envoy Redis filter) and designing complex, multi-tiered caching architectures
- Distributed Systems Expertise: Deep domain knowledge in building and operating large-scale distributed systems