When models arrive on SageMaker weeks or even months after their launch on Bedrock, it can leave teams tempted to migrate from one service to another.
In mid-2024, Slack had made the switch… in part on this premise. In the backdrop, the recent integration of search and summarization features powered by LLMs.
SageMaker and the «plumbing work»
The initial architecture deployed on SageMaker involved an escrowed VPC. The LLMs ran there, allowing Slack to assure users that their data « [remained] in-house ».
To maximize availability, these containers were deployed across several AWS regions. That did not, however, resolve the scaling latency. Nor the intermittent availability of A100 and H100 GPUs, as well as the need to overprovision to meet SLAs.
Slack had eliminated the scaling problem with cron-based scheduling. And the overprovisioning issue by reserving capacity on demand.
From manual coordination and the «plumbing work», however, it ultimately shifted toward automated capacity management. Bedrock delivered that… on top of faster access to the latest models.
Despite Bedrock, capacity remained underutilized
Mid-2024, Bedrock met Slack’s security and compliance expectations (it had earned FedRAMP, a mid-level certification). Beyond the managed aspect (no longer needing to operate GPU instances individually), it enabled tuning compute to match workloads. Playing, in this case, between provisioned throughput and on-demand capacity.
The first option was used for latency-sensitive interactive features, such as channel summarization.
The second was used for predictable workloads with high peak demand, such as the daily recap.
This pairing helped reduce unused resources. Nevertheless, to ensure a good experience for certain features, the team had to align with a traffic peak, even if that peak occurred only in a single geographic region. This was the case for the daily brief, dubbed « vue Aujourd’hui ». Usage was higher in the United States than in Europe and Asia, leading to underutilization of capacity.
A lock-in effect with provisioned throughput
The provisioned throughput presented another pitfall: the need to commit to 1–6 months. In the fast-moving world of LLMs, that cadence limits the ability to keep pace. Slack often chose to wait for commitments to expire before switching models.
Against this backdrop, everything gradually migrated to on-demand infrastructure, with an intermediate overflow-based step. Bedrock proved effective for multi-region routing based on availability. But the shared-resource model produced variable service levels — a problem not encountered with provisioned throughput. It also raised a risk of concentration, i.e., exposure to outages across the global infrastructure.
Slack therefore built resilience. It established, for each AI feature, a hierarchy of models. A failover was orchestrated when certain indicators degraded (time to first token, cap on request throughput, customer satisfaction level…).
Multicloud… with normalization
This mechanism did not protect against exposure to global outages. Slack also observed that model catalogs differed between cloud providers. Two factors that motivated the shift to multicloud.
In early 2026, Slack AI officially expanded to Google Cloud. It asserts that adding Vertex AI’s catalog to Bedrock’s allowed it to improve its quality metrics by about 10% on « complex reasoning tasks ». And to cut latency by 67% on the “small” workloads (low token usage).
The multicloud approach also enabled more traffic-routing strategies, built around various circuit breakers and A/B testing. To tackle cold starts, it implemented passwordless authentication and an API normalization layer. It also required building a unified stack for monitoring, since the native dashboards of one cloud and the other left gaps. The cost-tracking challenge also arose, more acute when workloads shift dynamically across clouds. As for the engineers, they had to train on both ecosystems…
