The story behind sisyphus, a better java retry framework

sisyphus combines the advantages of spring-retry and gauva-retrying, and is also very flexible to use.

Today, let us take a look at the story behind Sisyphus.

Scenario import

Simple needs

Product Manager: Realize a service that queries user information according to conditions.

Xiao Ming: Okay. no problem.

Code

• UserService.java

public interface UserService {

    /**
     * 根据条件查询用户信息
     * @param condition 条件
     * @return User 信息
     */
    User queryUser(QueryUserCondition condition);

}

• UserServiceImpl.java

public class UserServiceImpl implements UserService {

    private OutService outService;

    public UserServiceImpl(OutService outService) {
        this.outService = outService;
    }

    @Override
    public User queryUser(QueryUserCondition condition) {
        outService.remoteCall();
        return new User();
    }

}

conversation

Project manager: This service sometimes fails, look at it.

Xiao Ming: OutService is an external RPC service, but it is sometimes unstable.

Project Manager: If the call fails, you can retry several times when calling. Go and see something related to retry

Retry

Retry

There are scenarios for retrying. Not all scenarios are suitable for retrying. For example, parameter verification is illegal, write operations, etc. (considering whether the write is idempotent) are not suitable for retrying.

Remote call timeout or sudden network interruption can be retried. In the microservice governance framework, it usually has its own retry and timeout configuration. For example, dubbo can set retries=1 and timeout=500. If the call fails, only one retry is performed. If the call fails after 500ms, the call fails.

For example, operations such as external RPC calls, or data storage, if an operation fails, can be retried multiple times to increase the possibility of a successful call .

V1.0 supports retry version

think

Xiaoming: I have other tasks at hand, and this one is quite simple. It takes 5 minutes to get him.

accomplish

• UserServiceRetryImpl.java

public class UserServiceRetryImpl implements UserService {

    @Override
    public User queryUser(QueryUserCondition condition) {
        int times = 0;
        OutService outService = new AlwaysFailOutServiceImpl();

        while (times < RetryConstant.MAX_TIMES) {
            try {
                outService.remoteCall();
                return new User();
            } catch (Exception e) {
                times++;

                if(times >= RetryConstant.MAX_TIMES) {
                    throw new RuntimeException(e);
                }
            }
        }

        return null;
    }

}

V1.1 proxy mode version

Easy to maintain

Project manager: I read your code. Although the function is implemented, try to write it as easy as possible to maintain.

Xiao Ming: Okay. (I thought to myself, is it necessary to write some notes or something?)

Agency model

Provide a proxy for other objects to control access to this object.

In some cases, an object is not suitable or can not directly refer to another object, and the proxy object can play an intermediary role between the client and the target object.

Its characteristic is that the agent and the delegated class have the same interface.

accomplish

Xiao Ming think've seen before proxy mode, thinking this way, less original code changes later want to change it is also convenient .

• UserServiceProxyImpl.java

public class UserServiceProxyImpl implements UserService {

    private UserService userService = new UserServiceImpl();

    @Override
    public User queryUser(QueryUserCondition condition) {
        int times = 0;

        while (times < RetryConstant.MAX_TIMES) {
            try {
                return userService.queryUser(condition);
            } catch (Exception e) {
                times++;

                if(times >= RetryConstant.MAX_TIMES) {
                    throw new RuntimeException(e);
                }
            }
        }
        return null;
    }

}

V1.2 dynamic proxy mode

Easy to expand

Project Manager: Xiao Ming, there is another method that has the same problem. You can also add retry.

Xiao Ming: Okay.

Xiaoming thought to himself, I was writing an agent, but I changed my mind and calmed down. What if there is a service and I have to try again?

• RoleService.java

public interface RoleService {

    /**
     * 查询
     * @param user 用户信息
     * @return 是否拥有权限
     */
    boolean hasPrivilege(User user);

}

Code

• DynamicProxy.java

public class DynamicProxy implements InvocationHandler {

    private final Object subject;

    public DynamicProxy(Object subject) {
        this.subject = subject;
    }

    @Override
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        int times = 0;

        while (times < RetryConstant.MAX_TIMES) {
            try {
                // 当代理对象调用真实对象的方法时，其会自动的跳转到代理对象关联的handler对象的invoke方法来进行调用
                return method.invoke(subject, args);
            } catch (Exception e) {
                times++;

                if (times >= RetryConstant.MAX_TIMES) {
                    throw new RuntimeException(e);
                }
            }
        }

        return null;
    }

    /**
     * 获取动态代理
     *
     * @param realSubject 代理对象
     */
    public static Object getProxy(Object realSubject) {
        //    我们要代理哪个真实对象，就将该对象传进去，最后是通过该真实对象来调用其方法的
        InvocationHandler handler = new DynamicProxy(realSubject);
        return Proxy.newProxyInstance(handler.getClass().getClassLoader(),
                realSubject.getClass().getInterfaces(), handler);
    }

}

• Test code

@Test
public void failUserServiceTest() {
        UserService realService = new UserServiceImpl();
        UserService proxyService = (UserService) DynamicProxy.getProxy(realService);

        User user = proxyService.queryUser(new QueryUserCondition());
        LOGGER.info("failUserServiceTest: " + user);
}


@Test
public void roleServiceTest() {
        RoleService realService = new RoleServiceImpl();
        RoleService proxyService = (RoleService) DynamicProxy.getProxy(realService);

        boolean hasPrivilege = proxyService.hasPrivilege(new User());
        LOGGER.info("roleServiceTest: " + hasPrivilege);
}

V1.3 dynamic proxy mode enhancement

dialogue

Project Manager: Xiao Ming, your dynamic proxy method is quite lazy, but some of our classes have no interfaces. You have to solve this problem.

Xiao Ming: Okay. (Who? Write services without defining interfaces)

• ResourceServiceImpl.java

public class ResourceServiceImpl {

    /**
     * 校验资源信息
     * @param user 入参
     * @return 是否校验通过
     */
    public boolean checkResource(User user) {
        OutService outService = new AlwaysFailOutServiceImpl();
        outService.remoteCall();
        return true;
    }

}

Bytecode technology

Xiao Ming looked at the information on the Internet, and there was a solution.

• CGLIB

CGLIB is a powerful, high-performance and high-quality code generation library used to extend JAVA classes and implement interfaces at runtime.

• javassist

javassist (Java Programming Assistant) makes Java bytecode operations easy.

It is a class library for editing bytecode in Java; it allows Java programs to define new classes at runtime and modify class files when the JVM loads them.

Unlike other similar bytecode editors, Javassist provides two levels of API: source level and bytecode level.

If users use source code-level APIs, they can edit class files without needing to understand the Java bytecode specification.

The entire API is designed using only the vocabulary of the Java language. You can even specify the inserted bytecode in the form of source text; Javassist compiles it dynamically.

On the other hand, the bytecode level API allows users to directly edit class files as other editors.

• ASM

ASM is a general Java bytecode manipulation and analysis framework.

It can be used to modify existing classes or dynamically generate classes directly in binary form.

ASM provides some general bytecode conversion and analysis algorithms, from which you can build custom complex conversion and code analysis tools.

ASM provides functions similar to other Java bytecode frameworks, but mainly focuses on performance.

Because its design and implementation are as small and fast as possible, it is very suitable for use in dynamic systems (of course, it can also be used in a static way, such as in a compiler).

accomplish

Xiao Ming took a look and chose to use CGLIB.

• CglibProxy.java

public class CglibProxy implements MethodInterceptor {

    @Override
    public Object intercept(Object o, Method method, Object[] objects, MethodProxy methodProxy) throws Throwable {
        int times = 0;

        while (times < RetryConstant.MAX_TIMES) {
            try {
                //通过代理子类调用父类的方法
                return methodProxy.invokeSuper(o, objects);
            } catch (Exception e) {
                times++;

                if (times >= RetryConstant.MAX_TIMES) {
                    throw new RuntimeException(e);
                }
            }
        }

        return null;
    }

    /**
     * 获取代理类
     * @param clazz 类信息
     * @return 代理类结果
     */
    public Object getProxy(Class clazz){
        Enhancer enhancer = new Enhancer();
        //目标对象类
        enhancer.setSuperclass(clazz);
        enhancer.setCallback(this);
        //通过字节码技术创建目标对象类的子类实例作为代理
        return enhancer.create();
    }

}

• test

@Test
public void failUserServiceTest() {
   UserService proxyService = (UserService) new CglibProxy().getProxy(UserServiceImpl.class);

   User user = proxyService.queryUser(new QueryUserCondition());
   LOGGER.info("failUserServiceTest: " + user);
}

@Test
public void resourceServiceTest() {
   ResourceServiceImpl proxyService = (ResourceServiceImpl) new CglibProxy().getProxy(ResourceServiceImpl.class);
   boolean result = proxyService.checkResource(new User());
   LOGGER.info("resourceServiceTest: " + result);
}

V2.0 AOP implementation

dialogue

Project Manager: Xiao Ming, I was thinking about a question recently. Different services should have different retry times. Because services have different requirements for stability.

Xiao Ming: Okay. (I thought, it’s been a week since the retry, and it’s Friday today.)

Before leaving work, Xiao Ming had been thinking about this question. It's just the weekend, take a moment to write a retry gadget.

Design ideas

• Technical Support

spring

java annotation

• Annotation definition

Annotations can be used on methods to define the number of retries required

• Annotation analysis

Intercept and specify the method that needs to be retried, parse the corresponding number of retries, and then perform the corresponding number of retries.

accomplish

• Retryable.java

@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface Retryable {

    /**
     * Exception type that are retryable.
     * @return exception type to retry
     */
    Class<? extends Throwable> value() default RuntimeException.class;

    /**
     * 包含第一次失败
     * @return the maximum number of attempts (including the first failure), defaults to 3
     */
    int maxAttempts() default 3;

}

• RetryAspect.java

@Aspect
@Component
public class RetryAspect {

    @Pointcut("execution(public * com.github.houbb.retry.aop..*.*(..)) &&" +
                      "@annotation(com.github.houbb.retry.aop.annotation.Retryable)")
    public void myPointcut() {
    }

    @Around("myPointcut()")
    public Object around(ProceedingJoinPoint point) throws Throwable {
        Method method = getCurrentMethod(point);
        Retryable retryable = method.getAnnotation(Retryable.class);

        //1. 最大次数判断
        int maxAttempts = retryable.maxAttempts();
        if (maxAttempts <= 1) {
            return point.proceed();
        }

        //2. 异常处理
        int times = 0;
        final Class<? extends Throwable> exceptionClass = retryable.value();
        while (times < maxAttempts) {
            try {
                return point.proceed();
            } catch (Throwable e) {
                times++;

                // 超过最大重试次数 or 不属于当前处理异常
                if (times >= maxAttempts ||
                        !e.getClass().isAssignableFrom(exceptionClass)) {
                    throw new Throwable(e);
                }
            }
        }

        return null;
    }

    private Method getCurrentMethod(ProceedingJoinPoint point) {
        try {
            Signature sig = point.getSignature();
            MethodSignature msig = (MethodSignature) sig;
            Object target = point.getTarget();
            return target.getClass().getMethod(msig.getName(), msig.getParameterTypes());
        } catch (NoSuchMethodException e) {
            throw new RuntimeException(e);
        }
    }

}

Use of method

• fiveTimes()

The current method is retried 5 times in total.
Retry condition: the service throws AopRuntimeExption

@Override
@Retryable(maxAttempts = 5, value = AopRuntimeExption.class)
public void fiveTimes() {
    LOGGER.info("fiveTimes called!");
    throw new AopRuntimeExption();
}

• Test log

2018-08-08 15:49:33.814  INFO  [main] com.github.houbb.retry.aop.service.impl.UserServiceImpl:66 - fiveTimes called!
2018-08-08 15:49:33.815  INFO  [main] com.github.houbb.retry.aop.service.impl.UserServiceImpl:66 - fiveTimes called!
2018-08-08 15:49:33.815  INFO  [main] com.github.houbb.retry.aop.service.impl.UserServiceImpl:66 - fiveTimes called!
2018-08-08 15:49:33.815  INFO  [main] com.github.houbb.retry.aop.service.impl.UserServiceImpl:66 - fiveTimes called!
2018-08-08 15:49:33.815  INFO  [main] com.github.houbb.retry.aop.service.impl.UserServiceImpl:66 - fiveTimes called!

java.lang.reflect.UndeclaredThrowableException
...

V3.0 spring-retry version

dialogue

When I came to the company on Monday, the project manager talked with Xiao Ming again.

Project manager: The number of retries is sufficient, but the retry should actually pay attention to strategy. For example, if the external call fails for the first time, you can wait for 5S for the next call. If it fails again, you can wait for 10s to call again. . .

Xiao Ming: I understand.

think

But today Monday, there are many other things to do.

Xiao Ming was thinking, there is no time to write this. See if there is any ready-made online.

spring-retry

Spring Retry provides declarative retry support for Spring applications. It is used for Spring batch processing, Spring integration, Apache Hadoop (etc.) Spring.

In a distributed system, in order to ensure the strong consistency of data distributed transactions, when you call the RPC interface or send MQ, take a retry operation in response to the possible network jitter request timeout. The most common retry method for everyone is MQ, but if MQ is not introduced in your project, it will be inconvenient.

There is another way, developers write their own retry mechanism, but most of them are not elegant enough.

Annotated use

• RemoteService.java

Retry condition: encounter RuntimeException

Number of retries: 3

Retry strategy: Wait for 5S when retrying, and then the time will be doubled.

Fuse mechanism: if all retries fail, call the recover() method.

@Service
public class RemoteService {

    private static final Logger LOGGER = LoggerFactory.getLogger(RemoteService.class);

    /**
     * 调用方法
     */
    @Retryable(value = RuntimeException.class,
               maxAttempts = 3,
               backoff = @Backoff(delay = 5000L, multiplier = 2))
    public void call() {
        LOGGER.info("Call something...");
        throw new RuntimeException("RPC调用异常");
    }

    /**
     * recover 机制
     * @param e 异常
     */
    @Recover
    public void recover(RuntimeException e) {
        LOGGER.info("Start do recover things....");
        LOGGER.warn("We meet ex: ", e);
    }

}

• test

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.NONE)
public class RemoteServiceTest {

    @Autowired
    private RemoteService remoteService;

    @Test
    public void test() {
        remoteService.call();
    }

}

• Log

2018-08-08 16:03:26.409  INFO 1433 --- [           main] c.g.h.r.spring.service.RemoteService     : Call something...
2018-08-08 16:03:31.414  INFO 1433 --- [           main] c.g.h.r.spring.service.RemoteService     : Call something...
2018-08-08 16:03:41.416  INFO 1433 --- [           main] c.g.h.r.spring.service.RemoteService     : Call something...
2018-08-08 16:03:41.418  INFO 1433 --- [           main] c.g.h.r.spring.service.RemoteService     : Start do recover things....
2018-08-08 16:03:41.425  WARN 1433 --- [           main] c.g.h.r.spring.service.RemoteService     : We meet ex: 

java.lang.RuntimeException: RPC调用异常
    at com.github.houbb.retry.spring.service.RemoteService.call(RemoteService.java:38) ~[classes/:na]
...

The time point of the three calls:

2018-08-08 16:03:26.409 
2018-08-08 16:03:31.414
2018-08-08 16:03:41.416

defect

Although the spring-retry tool can implement retry gracefully, there are two unfriendly designs:

One is that the retry entity is limited to the Throwable subclass, indicating that the retry is designed for a catchable functional abnormality, but we hope to rely on a certain data object entity as the retry entity.
But the sping-retry framework must be cast to a subclass of Throwable.

The other is that the assertion object at the root of the retry uses the Exception instance of doWithRetry, which does not conform to the return design of the normal internal assertion.

Spring Retry advocates retrying methods in the form of annotations. The retry logic is executed synchronously. The "failure" of the retry is for Throwable.
If you want to determine whether you need to retry based on a certain state of the return value, you may only have to judge the return value by yourself and then explicitly throw an exception.

@Recover annotation cannot specify a method when it is used. If there are multiple retry methods in a class, it will be very troublesome.

guava-retrying

conversation

Xiaohua: Our system also needs to retry

Project manager: Xiaoming used spring-retry some time ago, it should be good to share

Xiaoming: Spring-retry has basic functions, but must be based on exceptions to control . If you want to determine whether you need to retry based on a certain state of the return value, you may only have to judge the return value by yourself and then explicitly throw an exception.

Xiaohua: In our project, we want to retry based on the properties of the object. You can look at guava-retry, I used it a long time ago and it feels pretty good.

Xiao Ming: Okay.

guava-retrying

guava-retrying The module provides a general method to retry arbitrary Java code using the specific stop, retry, and exception handling functions enhanced by Guava predicate matching.

• Advantage

The guava retryer tool is similar to spring-retry, which wraps the normal logic retry by defining the role of the retryer, but Guava retryer has a better definition of strategy, which is compatible on the basis of supporting the number of retries and retry frequency control. Supports the definition of retry sources for multiple exceptions or custom entity objects, allowing more flexibility in the retry function.

Guava Retryer also thread-safe, entry call logic is used in java.util.concurrent.Callable of call() method

Code example

Introductory case

After encountering an exception, retry 3 times to stop

• HelloDemo.java

public static void main(String[] args) {
    Callable<Boolean> callable = new Callable<Boolean>() {
        @Override
        public Boolean call() throws Exception {
            // do something useful here
            LOGGER.info("call...");
            throw new RuntimeException();
        }
    };

    Retryer<Boolean> retryer = RetryerBuilder.<Boolean>newBuilder()
            .retryIfResult(Predicates.isNull())
            .retryIfExceptionOfType(IOException.class)
            .retryIfRuntimeException()
            .withStopStrategy(StopStrategies.stopAfterAttempt(3))
            .build();
    try {
        retryer.call(callable);
    } catch (RetryException | ExecutionException e) {
        e.printStackTrace();
    }

}

• Log

2018-08-08 17:21:12.442  INFO  [main] com.github.houbb.retry.guava.HelloDemo:41 - call...
com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 3 attempts.
2018-08-08 17:21:12.443  INFO  [main] com.github.houbb.retry.guava.HelloDemo:41 - call...
2018-08-08 17:21:12.444  INFO  [main] com.github.houbb.retry.guava.HelloDemo:41 - call...
    at com.github.rholder.retry.Retryer.call(Retryer.java:174)
    at com.github.houbb.retry.guava.HelloDemo.main(HelloDemo.java:53)
Caused by: java.lang.RuntimeException
    at com.github.houbb.retry.guava.HelloDemo$1.call(HelloDemo.java:42)
    at com.github.houbb.retry.guava.HelloDemo$1.call(HelloDemo.java:37)
    at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
    at com.github.rholder.retry.Retryer.call(Retryer.java:160)
    ... 1 more

Summarize

Graceful retry commonality and principle

Normal and retry are gracefully decoupled, and retry assertion condition instances or logic exception instances are the communication medium between the two.

Agreed retry intervals, differentiated retry strategies, and set retry timeouts to further ensure the effectiveness of retry and the stability of the retry process.

All use the command design pattern, complete the corresponding logic operation by entrusting the retry object, and internally encapsulate the retry logic.

Both spring-retry and guava-retry tools are thread-safe retries and can support the correctness of retry logic in concurrent business scenarios.

Graceful retry applicable scenarios

There are unstable dependent scenarios in the functional logic, and you need to use retry to obtain the expected result or try to re-execute the logic without ending immediately. Such as remote interface access, data loading access, data upload verification and so on.

For abnormal scenarios, there are scenarios that require retry, and at the same time, it is hoped that normal logic and retry logic can be decoupled.

For scenarios that require data-based interactions and wish to execute logic scenarios through retry polling detection, retry solutions can also be considered.

conversation

Project manager: I think guava-retry is pretty good, but it's not convenient enough. Xiaoming, you can encapsulate an annotation-based one.

Xiao Ming:...

Better realization

So Xiao Ming wrote sisyphus in tears.

java retry framework-sisyphus

I hope this article is helpful to you. If you like it, please like, collect and forward a wave.

I am an old horse, and I look forward to seeing you again next time.