Sentinel 是怎樣攔截異常流量的？
各位在家里用電的過(guò)程中，一定也經(jīng)歷過(guò)「跳閘」。這個(gè)「閘」就是在電量超過(guò)負(fù)荷的時(shí)候用來(lái)保護(hù)我們用電安全的，也被稱為「斷路器」，還有個(gè)響亮的英文名 -- CircuitBreaker。
和用電安全一樣，對(duì)于「限流」、「降級(jí)」、「熔斷」...，你我應(yīng)該也都耳熟能詳。我們開發(fā)的各類軟件、系統(tǒng)、互聯(lián)網(wǎng)應(yīng)用等為了不被異常流量壓垮，也需要一個(gè)斷路器。
在 Spring 應(yīng)用中，使用斷路器很方便，我們可以使用 Spring Cloud CircuitBreaker。
Spring Cloud Circuit Breaker 是啥?如果你熟悉 Spring 是什么人的話，你能猜個(gè)八九不離十。和Spring Data JPA 這些類似，Spring 他又搞了個(gè)抽象的，標(biāo)準(zhǔn)的API 出來(lái)。這次他抽象的是關(guān)于降級(jí)熔斷的「斷路器」。有了這一層，具體實(shí)現(xiàn)是誰(shuí)可以方便的更換，我們使用的代碼里改動(dòng)基本為0。
我們先來(lái)從官方Demo有個(gè)初步印象：
- @RestController
- public class DemoController {
- private CircuitBreakerFactory circuitBreakerFactory;
- private HttpBinService httpBin;
- public DemoController(CircuitBreakerFactory circuitBreakerFactory, HttpBinService httpBinService) {
- this.circuitBreakerFactory = circuitBreakerFactory;
- this.httpBin = httpBinService;
- }
- @GetMapping("/delay/{seconds}")
- public Map delay(@PathVariable int seconds) {
- return circuitBreakerFactory.create("delay").run(httpBin.delaySuppplier(seconds), t -> {
- Map<String, String> fallback = new HashMap<>();
- fallback.put("hello", "world");
- return fallback;
- });
- }
- }
千言萬(wàn)語(yǔ)，總結(jié)出來(lái)這樣一句circuitBreakerFactory.create("delay").run()
因?yàn)槭浅橄螅瑢?duì)應(yīng)的實(shí)現(xiàn)就有好多種啦。
目前支持的實(shí)現(xiàn)有：
- Hystrix
- Resilience4j
- Sentinel
- Spring Retry
而抽象相當(dāng)于定了個(gè)標(biāo)準(zhǔn)，像JDBC一樣，無(wú)論我們把數(shù)據(jù)庫(kù)換成了MySQL，Oracle 還是SQLite，接口等非特定類型的代碼都不需要改變。斷路器也一樣。
這里的斷路器工廠，創(chuàng)建方法都是標(biāo)準(zhǔn)的。具體這里執(zhí)行業(yè)務(wù)邏輯的時(shí)候斷路器實(shí)現(xiàn)要怎樣進(jìn)行攔截降級(jí)，就可以交給具體的實(shí)現(xiàn)來(lái)完成。
這次，我們以開源的 Sentinel 為例，來(lái)看看他們是怎樣攔住異常流量的。
首先，因?yàn)槭荢pring Cloud，所以還會(huì)基于 Spring Boot 的 Autoconfiguration。以下是配置類，我們看到生成了一個(gè)工廠。
- public class SentinelCircuitBreakerAutoConfiguration {
- @Bean
- @ConditionalOnMissingBean(CircuitBreakerFactory.class)
- public CircuitBreakerFactory sentinelCircuitBreakerFactory() {
- return new SentinelCircuitBreakerFactory();
- }
- }
在我們實(shí)際代碼執(zhí)行邏輯的時(shí)候，create 出來(lái)的是什么呢?
是個(gè)斷路器 CircuitBreaker，用來(lái)執(zhí)行代碼。
- public interface CircuitBreaker {
- default <T> T run(Supplier<T> toRun) {
- return run(toRun, throwable -> {
- throw new NoFallbackAvailableException("No fallback available.", throwable);
- });
- };
- <T> T run(Supplier<T> toRun, Function<Throwable, T> fallback);
- }
包含兩個(gè)執(zhí)行的方法，需要在的時(shí)候可以指定fallback邏輯。具體到 Sentinel 是這樣的：
- public CircuitBreaker create(String id) {
- SentinelConfigBuilder.SentinelCircuitBreakerConfiguration conf = getConfigurations()
- .computeIfAbsent(id, defaultConfiguration);
- return new SentinelCircuitBreaker(id, conf.getEntryType(), conf.getRules());
- }
你會(huì)看到創(chuàng)建了一個(gè)SentinelCircuitBreaker。我們的業(yè)務(wù)邏輯，就會(huì)在這個(gè)斷路器里執(zhí)行,run方法就是各個(gè)具體實(shí)現(xiàn)的舞臺(tái)。
- @Override
- public <T> T run(Supplier<T> toRun, Function<Throwable, T> fallback) {
- Entry entry = null;
- try {
- entry = SphU.entry(resourceName, entryType);
- // If the SphU.entry() does not throw `BlockException`, it means that the
- // request can pass.
- return toRun.get();
- }
- catch (BlockException ex) {
- // SphU.entry() may throw BlockException which indicates that
- // the request was rejected (flow control or circuit breaking triggered).
- // So it should not be counted as the business exception.
- return fallback.apply(ex);
- }
- catch (Exception ex) {
- // For other kinds of exceptions, we'll trace the exception count via
- // Tracer.trace(ex).
- Tracer.trace(ex);
- return fallback.apply(ex);
- }
- finally {
- // Guarantee the invocation has been completed.
- if (entry != null) {
- entry.exit();
- }
- }
- }
OK，到此為止， Spring Cloud CircuitBreaker 已經(jīng)展現(xiàn)完了。其它的細(xì)節(jié)都放到了具體實(shí)現(xiàn)的「盒子」里。下面我們把這個(gè)盒子打開。
Sentinel 是個(gè)熔斷降級(jí)框架，官方這樣自我介紹：
面向分布式服務(wù)架構(gòu)的高可用流量控制組件,主要以流量為切入點(diǎn)，從流量控制、熔斷降級(jí)、系統(tǒng)自適應(yīng)保護(hù)等多個(gè)維度來(lái)幫助用戶保障微服務(wù)的穩(wěn)定性。
官網(wǎng)的這張代碼截圖簡(jiǎn)潔的說(shuō)明了他是怎樣工作的
擋在業(yè)務(wù)代碼的前面，有事兒先沖它來(lái)，能通過(guò)之后才走業(yè)務(wù)邏輯，和各類闖關(guān)還真類似。
在上面 CircuitBreaker 的 run 方法里，咱們一定都注意到了這句
- entry = SphU.entry(resourceName, entryType);
這就是一切攔截的秘密。
無(wú)論我們是通過(guò)前面的CircuitBreaker的方式，還是 @SentinelResource 這種注解形式，還是通過(guò) Interceptor 的方式，沒(méi)什么本質(zhì)區(qū)別。只是觸發(fā)點(diǎn)不一樣。最后都是通過(guò)SphU來(lái)搞定。
既然是攔截，那一定要攔下來(lái)做這樣或那樣的檢查。
實(shí)際檢查的時(shí)候，entry 里核心代碼有這些：
- Entry entryWithPriority(ResourceWrapper resourceWrapper, ...)
- throws BlockException {
- ProcessorSlot<Object> chain = lookProcessChain(resourceWrapper);
- Entry e = new CtEntry(resourceWrapper, chain, context);
- try {
- chain.entry(context, resourceWrapper,...);
- } catch (BlockException e1) {
- e.exit(count, args);
- throw e1;
- }
- return e;
- }
注意這里的ProcessorSlot chain = lookProcessChain(resourceWrapper);會(huì)在請(qǐng)求過(guò)來(lái)處理的時(shí)候，如果未初始化處理鏈，則進(jìn)行初始化，將各種first,next設(shè)置好，后面的請(qǐng)求都會(huì)按這個(gè)來(lái)處理。所有需要攔截的Slot，都會(huì)加到這個(gè) chain 里面，再逐個(gè)執(zhí)行 chain 里的 slot。和Servlet Filter 類似。
chain里都加了些啥呢?
- public class HotParamSlotChainBuilder implements SlotChainBuilder {
- public ProcessorSlotChain build() {
- ProcessorSlotChain chain = new DefaultProcessorSlotChain();
- chain.addLast(new NodeSelectorSlot());
- chain.addLast(new ClusterBuilderSlot());
- chain.addLast(new LogSlot());
- chain.addLast(new StatisticSlot());
- chain.addLast(new ParamFlowSlot());
- chain.addLast(new SystemSlot());
- chain.addLast(new AuthoritySlot());
- chain.addLast(new FlowSlot());
- chain.addLast(new DegradeSlot());
- return chain;
- }
初始的時(shí)候，first 指向一個(gè)匿名內(nèi)部類，這些加進(jìn)來(lái)的slot，會(huì)在每次addLast的時(shí)候，做為鏈的next，
- AbstractLinkedProcessorSlot<?> end = first;
- @Override
- public void addFirst(AbstractLinkedProcessorSlot<?> protocolProcessor) {
- protocolProcessor.setNext(first.getNext());
- first.setNext(protocolProcessor);
- if (end == first) {
- end = protocolProcessor;
- }
- }
- @Override
- public void addLast(AbstractLinkedProcessorSlot<?> protocolProcessor) {
- end.setNext(protocolProcessor);
- end = protocolProcessor;
- }
而每個(gè) slot，有自己的特定用處，處理完自己的邏輯之后，會(huì)通過(guò) fireEntry 來(lái)觸發(fā)下一個(gè) slot的執(zhí)行。
給你一張長(zhǎng)長(zhǎng)的線程調(diào)用棧就會(huì)過(guò)分的明顯了：
- java.lang.Thread.State: RUNNABLE
- at com.alibaba.csp.sentinel.slots.block.flow.FlowSlot.checkFlow(FlowSlot.java:168)
- at com.alibaba.csp.sentinel.slots.block.flow.FlowSlot.entry(FlowSlot.java:161)
- at com.alibaba.csp.sentinel.slots.block.flow.FlowSlot.entry(FlowSlot.java:139)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.transformEntry(AbstractLinkedProcessorSlot.java:40)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.fireEntry(AbstractLinkedProcessorSlot.java:32)
- at com.alibaba.csp.sentinel.slots.block.authority.AuthoritySlot.entry(AuthoritySlot.java:39)
- at com.alibaba.csp.sentinel.slots.block.authority.AuthoritySlot.entry(AuthoritySlot.java:33)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.transformEntry(AbstractLinkedProcessorSlot.java:40)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.fireEntry(AbstractLinkedProcessorSlot.java:32)
- at com.alibaba.csp.sentinel.slots.system.SystemSlot.entry(SystemSlot.java:36)
- at com.alibaba.csp.sentinel.slots.system.SystemSlot.entry(SystemSlot.java:30)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.transformEntry(AbstractLinkedProcessorSlot.java:40)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.fireEntry(AbstractLinkedProcessorSlot.java:32)
- at com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowSlot.entry(ParamFlowSlot.java:39)
- at com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowSlot.entry(ParamFlowSlot.java:33)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.transformEntry(AbstractLinkedProcessorSlot.java:40)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.fireEntry(AbstractLinkedProcessorSlot.java:32)
- at com.alibaba.csp.sentinel.slots.statistic.StatisticSlot.entry(StatisticSlot.java:57)
- at com.alibaba.csp.sentinel.slots.statistic.StatisticSlot.entry(StatisticSlot.java:50)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.transformEntry(AbstractLinkedProcessorSlot.java:40)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.fireEntry(AbstractLinkedProcessorSlot.java:32)
- at com.alibaba.csp.sentinel.slots.logger.LogSlot.entry(LogSlot.java:35)
- at com.alibaba.csp.sentinel.slots.logger.LogSlot.entry(LogSlot.java:29)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.transformEntry(AbstractLinkedProcessorSlot.java:40)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.fireEntry(AbstractLinkedProcessorSlot.java:32)
- at com.alibaba.csp.sentinel.slots.clusterbuilder.ClusterBuilderSlot.entry(ClusterBuilderSlot.java:101)
- at com.alibaba.csp.sentinel.slots.clusterbuilder.ClusterBuilderSlot.entry(ClusterBuilderSlot.java:47)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.transformEntry(AbstractLinkedProcessorSlot.java:40)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.fireEntry(AbstractLinkedProcessorSlot.java:32)
- at com.alibaba.csp.sentinel.slots.nodeselector.NodeSelectorSlot.entry(NodeSelectorSlot.java:171)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.transformEntry(AbstractLinkedProcessorSlot.java:40)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.fireEntry(AbstractLinkedProcessorSlot.java:32)
- at com.alibaba.csp.sentinel.slotchain.DefaultProcessorSlotChain$1.entry(DefaultProcessorSlotChain.java:31)
- at com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot.transformEntry(AbstractLinkedProcessorSlot.java:40)
- at com.alibaba.csp.sentinel.slotchain.DefaultProcessorSlotChain.entry(DefaultProcessorSlotChain.java:75)
- at com.alibaba.csp.sentinel.CtSph.entryWithPriority(CtSph.java:148)
- at com.alibaba.csp.sentinel.CtSph.entryWithType(CtSph.java:347)
- at com.alibaba.csp.sentinel.CtSph.entryWithType(CtSph.java:340)
- at com.alibaba.csp.sentinel.SphU.entry(SphU.java:285)
降級(jí)有三種類型
每種類型，都會(huì)根據(jù)對(duì)應(yīng)的配置項(xiàng)數(shù)據(jù)比對(duì)，不符合就中斷，中斷之后也不能一直斷著，啥時(shí)候再恢復(fù)呢?就根據(jù)配置的時(shí)間窗口，會(huì)啟動(dòng)一個(gè)恢復(fù)線程，到時(shí)間就會(huì)調(diào)度，把中斷標(biāo)識(shí)恢復(fù)。
- public boolean passCheck(Context context, DefaultNode node, int acquireCount, Object... args) {
- if (cut.get()) {
- return false;
- }
- ClusterNode clusterNode = ClusterBuilderSlot.getClusterNode(this.getResource());
- if (clusterNode == null) {
- return true;
- }
- if (grade == RuleConstant.DEGRADE_GRADE_RT) {
- double rt = clusterNode.avgRt();
- if (rt < this.count) {
- passCount.set(0);
- return true;
- }
- // Sentinel will degrade the service only if count exceeds.
- if (passCount.incrementAndGet() < rtSlowRequestAmount) {
- return true;
- }
- } else if (grade == RuleConstant.DEGRADE_GRADE_EXCEPTION_RATIO) {
- double exception = clusterNode.exceptionQps();
- double success = clusterNode.successQps();
- double total = clusterNode.totalQps();
- // If total amount is less than minRequestAmount, the request will pass.
- if (total < minRequestAmount) {
- return true;
- }
- // In the same aligned statistic time window,
- // "success" (aka. completed count) = exception count + non-exception count (realSuccess)
- double realSuccess = success - exception;
- if (realSuccess <= 0 && exception < minRequestAmount) {
- return true;
- }
- if (exception / success < count) {
- return true;
- }
- } else if (grade == RuleConstant.DEGRADE_GRADE_EXCEPTION_COUNT) {
- double exception = clusterNode.totalException();
- if (exception < count) {
- return true;
- }
- }
- if (cut.compareAndSet(false, true)) {
- ResetTask resetTask = new ResetTask(this);
- pool.schedule(resetTask, timeWindow, TimeUnit.SECONDS);
- }
- return false;
- }
恢復(fù)做了兩件事：一、把passCount設(shè)置成0，二、中斷標(biāo)識(shí)還原
上面介紹了對(duì)請(qǐng)求的攔截處理，這其中最核心的，也就是我們最主要配置的，一個(gè)是「流控」，一個(gè)是「降級(jí)」。這兩個(gè)對(duì)應(yīng)的Slot，會(huì)在處理請(qǐng)求的時(shí)候，根據(jù)配置好的 「規(guī)則」rule 來(lái)判斷。比如我們上面看到的時(shí)間窗口、熔斷時(shí)間等，以及流控的線程數(shù)，QPS數(shù)這些。
這些規(guī)則默認(rèn)的配置在內(nèi)存里，也可以通過(guò)不同的數(shù)據(jù)源加載進(jìn)來(lái)。同時(shí)啟用了Sentinel 控制臺(tái)的話，在控制臺(tái) 也可以配置規(guī)則。這些規(guī)則，會(huì)通過(guò) HTTP 發(fā)送給對(duì)應(yīng)使用了 sentinel 的應(yīng)用實(shí)例節(jié)點(diǎn)。
本文轉(zhuǎn)載自微信公眾號(hào)「 Tomcat那些事兒」，可以通過(guò)以下二維碼關(guān)注。轉(zhuǎn)載本文請(qǐng)聯(lián)系 Tomcat那些事兒公眾號(hào)。
