Connect with us


Incident Post Mortem: November 23, 2021




Between 4:00 pm and approximately 5:36 pm PT on Tuesday, November 23rd, we experienced an outage across most Coinbase production systems. During this outage, users were unable to access Coinbase using our websites and apps, and therefore were unable to use our products. This post is intended to describe what occurred and the causes, and to discuss how we plan to avoid such problems in the future.

The Incident

On November 23rd, 2021, at 4:00pm PT (Nov 24, 2021 00:00 UTC) an SSL certificate for an internal hostname in one of our Amazon Web Services (AWS) accounts expired. The expired SSL certificate was used by many of our internal load balancers which caused a majority of inter-service communications to fail. Due to the fact that our API routing layer connects to backend services via subdomains of this internal hostname, about 90% of incoming API traffic returned errors.

Error rates returned to normal once we were able to migrate all load balancers to a valid certificate.

Chart depicting overall 90% error rate at our API routing layer for duration of incident.

Context: Certificates at Coinbase

It’s helpful to provide some background information about how we manage SSL certificates at Coinbase. For the most part, certificates for public hostnames like are managed and provisioned by Cloudflare. For certificates for internal hostnames used to route traffic between backend services, we historically leveraged AWS IAM Server Certificates.

One of the downsides of IAM Server Certificates is that certificates must be generated outside of AWS and uploaded via an API call. So last year, our infrastructure team migrated from IAM Server Certificates to AWS Certificate Manager (ACM). ACM solves the security problem because AWS generates both the public and private components of the certificate within ACM and stores the encrypted version in IAM for us. Only connected services like Cloudfront and Elastic Load Balancers will get access to the certificates. Denying the acm:ExportCertificate permission to all AWS IAM Roles ensures that they can’t be exported.

In addition to the added security benefits, ACM also automatically renews certificates before expiration. Given that ACM certificates are supposed to renew and we did a migration, how did this happen?

Root Cause Analysis

Incident responders quickly noticed that the expired certificate was an IAM Server Certificate. This was unexpected because the aforementioned ACM migration had been widely publicized in engineering communication channels at the time; thus we had been operating under the assumption that we were running exclusively on ACM certificates.

As we later discovered, one of the certificate migrations didn’t go as planned; the group of engineers working on the migration uploaded a new IAM certificate and postponed the rest of the migration. Unfortunately, the delay was not as widely communicated as it should have been and changes to team structure and personnel resulted in the project being incorrectly assumed complete.

Migration status aside, you may ask the same question we asked ourselves: “Why weren’t we alerted to this expiring certificate?” The answer is: we were. Alerts were being sent to an email distribution group that we discovered only consisted of two individuals. This group was originally larger, but shrank with the departure of team members and was never sufficiently repopulated as new folks joined the team.

In short, the critical certificate was allowed to expire due all of three factors:

  1. The IAM to ACM migration was incomplete.
  2. Expiration alerts were only being sent via email and were filtered or ignored.
  3. Only two individuals were on the email distribution list.

Resolution & Improvements

In order to resolve the incident we migrated all of the load balancers that were using the expired IAM cert to the existing auto-renewing ACM cert that had been provisioned as part of the original migration plan. This took longer than desired due to the number of load balancers involved and our cautiousness in defining, testing, and applying the required infrastructure changes.

In order to ensure we don’t run into an issue like this again, we’ve taken the following steps to address the factors mentioned in the RCA section above:

  1. We’ve completed the migration to ACM, are no longer using IAM Server Certificates and are deleting any legacy certificates to reduce noise.
  2. We’re adding automated monitoring that is connected to our alerting and paging system to augment the email alerts. These will page on impending expiration as well as when ACM certificates drop out of auto-renewal eligibility.
  3. We’ve added a permanent group-alias to the email distribution list. Furthermore, this group is automatically updated as employees join and leave the company.
  4. We’re building a repository of incident remediation operations in order to reduce time to define, test and apply new changes.

We take the uptime and performance of our infrastructure very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency. If you’re interested in solving challenges like those listed here, come work with us.

Incident Post Mortem: November 23, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source link

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published.


Why Gold Is Beating Bitcoin In 2022



Bitcoin continues to underperform as a general “risk-off” sentiment has investors driving toward gold as a safe haven asset.

Not Risking It

Concerns about the Russo-Ukrainian war continue. The U.S. inflation struggles at a four-decade high and Fed rate hike fears prevail. The uncertainty extends to the world economy as a recession is expected instead of a recovery. The IMF’s managing director Kristalina Georgieva called it “a crisis on top of a crisis.”

“The war is a supply shock that reduces economic output and raises prices. Indeed, we forecast inflation will accelerate to 5.5 percent in advanced economies and to 9.3 percent in emerging European economies excluding Russia, Turkey, and Ukraine. ” The IMF stated last week.

Reuters recently quoted Commerzbank analyst Daniel Briesemann, who talked in a note about the factors that have “lent buoyancy to gold in recent days,” mentioning the “strong buying interest on the part of ETF (Exchange Traded Fund) investors” and news about the Ukraine war.

“Russia appears to be preparing to launch a major offensive in the east of the country – that is generating considerable demand for gold as a safe haven,” the analyst said.

This summarizes the “risk-off” sentiment at the moment. As expected, equities suffer as investors are selling risky assets and purchasing the ones negatively correlated to the traditional market. Thus, the crypto space is struggling alongside de stocks market and gold is rising.

Bitcoin Outperformed By Gold

Data from Arcane Research’s latest weekly report notes that it has been a gloomy year for the “digital gold.” In the first three weeks of 2022, Bitcoin sank 25% and it is still down by 18% in the year despite its slight recovery.

Similarly, Nasdaq records a 19% decline in the year, having underperformed against bitcoin “by a small margin,” notes the report, adding that “This is surprising given that bitcoin has tended to follow Nasdaq, albeit with higher volatility.”

The general fear over geopolitical and macroeconomic uncertainty has given gold the safe-haven asset spotlight once more. The asset outperformed all the other indexes seen below with a 4% gain.

Physical gold outperforming “digital gold” in 2022 | Source: Arcane Research

Meanwhile, the currency market is performing with “the same risk-off patterns.” The Dollar has been proving its “risk-off” dominance as the US Dollar Index (DXY) is up 7%. The Chinese yuan has taken a hit over concerns about the country’s “zero-covid” policy –which creates issues for the global supply chain– and the slowing down Chinese economy. In contrast, investors have been running to the US Dollar for safety.

Bitcoin supporters usually refer to the coin as “digital gold” alleging it is a safe haven asset, and this narrative had held well while BTC had been “uncorrelated with most other major asset classes,” but the tide is shifting with the 2022 scenario as investors are rather placing the coin “into the risk-on basket”.

A previous Arcane Research report indicated that bitcoin’s 30 -day correlation with the Nasdaq is revisiting July 2020 highs while its correlation with gold has reached all-time lows.

A pseudonym traded noted that “As Bitcoin adoption goes on and more institutional investors enter the market, the correlation of BTC and stocks becomes more and more tight. That is a paradigm that the crypto world struggled to come to terms with in the past but is now more real than ever. A healthy stock market is good for Bitcoin.”

Meanwhile, the general sentiment of traders seems to be bearish, with many saying that the coin could visit the $30k level soon.

Bitcoin trading at $39k in the daily chart | BTCUSD on

Source link

Continue Reading


Attendees talk the future of NFTs



The crypto community headed to Nassau in the Bahamas this week for the inaugural Crypto Bahamas conference.

Like most conferences, panels fill up the agenda and on Wednesday the topics at Crypto Bahamas ranged from NFTs to crypto in sports and to asset allocation in Web3. During one particular conversation, titled Evolution of NFTs: Culture, Utility and Regulation, panelists had some insightful musings on the NFT market.

To put the Crypto Bahamas conference into context, Sam Bankman-Fried’s cryptocurrency exchange FTX moved its headquarters from Hong Kong to the Bahamas in Sept. 2021. It recently inked a multi-year partnership with Anthony Scaramucci’s investment firm SkyBridge Capital, and its events arm SkyBridge Alternatives, or SALT. They jointly presented the conference.

That’s why the NFT panel consisted of multiple perspectives from Tristan Yver, head of strategy at FTX U.S., Joseph Doll, attorney at Fenwick law firm, Roham Gharegozlou, the chief executive officer at Dapper Labs, and Sarah Hammer, the managing director of The Stevens Center for Innovation in Finance at The Wharton School. Zack Guzman, writer for the Meta-owned newsletter platform Bulletin, moderated.

Gharegozlou pointed out how new the NFT market truly is when “most people have only been thinking about it for a year and a half,” making valuations “very immature.” As the CEO of Dapper Labs, the company behind NBA Top Shot,  Gharegozlou recognized that “utility, rewards and the how you value and NFT is primarily based on the strength of that of the community.”

He added that a good way for an NFT collection to build a strong community is to have multiple tiers of scarcity. In the case of NBA Top Shot, at the higher price end there is extreme scarcity, but there are also millions of “common” moments so that people can “get their first NFT and see how it feels without breaking the bank.” 

Tristan Yver echoed that the current valuation and pricing model for NFTs is based on a collective perception on value based on the amount of people willing to buy an asset for a certain amount. He anticipated a “movement away from this consensus view to a more unique singular view where people buy things that resonate with them rather than what resonates with a larger community.”

Joseph Doll chimed in to say that “communities need to be thoughtful about democratizing access.” There are some “massive” barriers to entry to certain projects, he said, including not being early enough or not having enough capital at the time. He questioned, “That’s not what crypto is about, right? It’s kind of about the exact opposite of that.” Democratization, he suggested, can come in the form of derivative projects at better price points.

Another important point brought up by Yver was the reality of scams, especially on Discord and Twitter. He said that “we need to move past security aspects to be able to really bring in the next large mass of users.” He recommended talking among family and friends or asking a Discord moderator to make sure “you click the right link when minting that NFT” because “wallet security sucks right now.”

Gharegozlou even said that Elon Musk, the new owner of Twitter, should use Web3 to fix Twitter’s fraud problem, just as Discord should use Web3 authentication and verification as well. “Once NFT’s are the sort of identity bridge across all these different social networks, identity and assets, authenticity, provenance,” then the system can be more resilient he added.

When asked what “main alpha” the audience should bear in mind, Doll said to engage with and be part of these NFT communities even if it’s “scary,” because getting scammed is a “part of the journey.”

Sarah Hammer, who leads the Cypher Accelerator at Wharton business school, said that the school is launching an incubator specifically for NFT projects in partnership with Dapper Labs because the “NFT model is a business model for the future.” She emphasized that the greatest way to grow and innovate in the space is to increase education efforts in order to get more people learning and working together.

Related: Goldman Sachs reportedly eyes FTX alliance with regulatory and public listing assistance

Recently the Bahamian government allowed residents to use digital assets, including the world’s first central bank digital currency, or CBDC, to pay for taxes in 2022.