I: pbuilder: network access will be disabled during build
I: Current time: Wed May  8 08:30:18 +14 2024
I: pbuilder-time-stamp: 1715106618
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
gpgv: Signature made Thu Jul 27 15:15:46 2023 
gpgv:                using RSA key 303F687A4EFCB1AE7C1A35A3B700173FBB805A29
gpgv:                issuer "dswarbrick@debian.org"
gpgv: Can't check signature: No public key
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/2764266/tmp/hooks/D01_modify_environment starting
debug: Running on infom01-amd64.
I: Changing host+domainname to test build reproducibility
I: Adding a custom variable just for the fun of it...
I: Changing /bin/sh to bash
'/bin/sh' -> '/bin/bash'
lrwxrwxrwx 1 root root 9 May  7 18:30 /bin/sh -> /bin/bash
I: Setting pbuilder2's login shell to /bin/bash
I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other
I: user script /srv/workspace/pbuilder/2764266/tmp/hooks/D01_modify_environment finished
I: user script /srv/workspace/pbuilder/2764266/tmp/hooks/D02_print_environment starting
I: set
  BASH=/bin/sh
  BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath
  BASH_ALIASES=()
  BASH_ARGC=()
  BASH_ARGV=()
  BASH_CMDS=()
  BASH_LINENO=([0]="12" [1]="0")
  BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:.
  BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment")
  BASH_VERSINFO=([0]="5" [1]="2" [2]="21" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
  BASH_VERSION='5.2.21(1)-release'
  BUILDDIR=/build/reproducible-path
  BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other'
  BUILDUSERNAME=pbuilder2
  BUILD_ARCH=amd64
  DEBIAN_FRONTEND=noninteractive
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 '
  DIRSTACK=()
  DISTRIBUTION=trixie
  EUID=0
  FUNCNAME=([0]="Echo" [1]="main")
  GROUPS=()
  HOME=/root
  HOSTNAME=i-capture-the-hostname
  HOSTTYPE=x86_64
  HOST_ARCH=amd64
  IFS=' 	
  '
  INVOCATION_ID=0ec4bb30d07242c9a4541eca85152c23
  LANG=C
  LANGUAGE=et_EE:et
  LC_ALL=C
  MACHTYPE=x86_64-pc-linux-gnu
  MAIL=/var/mail/root
  OPTERR=1
  OPTIND=1
  OSTYPE=linux-gnu
  PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path
  PBCURRENTCOMMANDLINEOPERATION=build
  PBUILDER_OPERATION=build
  PBUILDER_PKGDATADIR=/usr/share/pbuilder
  PBUILDER_PKGLIBDIR=/usr/lib/pbuilder
  PBUILDER_SYSCONFDIR=/etc
  PIPESTATUS=([0]="0")
  POSIXLY_CORRECT=y
  PPID=2764266
  PS4='+ '
  PWD=/
  SHELL=/bin/bash
  SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix
  SHLVL=3
  SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.oVBWptuN/pbuilderrc_rP8G --distribution trixie --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.oVBWptuN/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID=109
  SUDO_UID=104
  SUDO_USER=jenkins
  TERM=unknown
  TZ=/usr/share/zoneinfo/Etc/GMT-14
  UID=0
  USER=root
  _='I: set'
I: uname -a
  Linux i-capture-the-hostname 6.1.0-21-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 May  5 11:23 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/2764266/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: amd64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19698 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.22-go{a} golang-1.22-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 
0 packages upgraded, 103 newly installed, 0 to remove and 0 not upgraded.
Need to get 90.2 MB of archives. After unpacking 626 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian trixie/main amd64 sensible-utils all 0.0.22 [22.4 kB]
Get: 2 http://deb.debian.org/debian trixie/main amd64 libmagic-mgc amd64 1:5.45-3 [314 kB]
Get: 3 http://deb.debian.org/debian trixie/main amd64 libmagic1t64 amd64 1:5.45-3 [105 kB]
Get: 4 http://deb.debian.org/debian trixie/main amd64 file amd64 1:5.45-3 [42.9 kB]
Get: 5 http://deb.debian.org/debian trixie/main amd64 gettext-base amd64 0.21-14+b1 [161 kB]
Get: 6 http://deb.debian.org/debian trixie/main amd64 libuchardet0 amd64 0.0.8-1+b1 [68.8 kB]
Get: 7 http://deb.debian.org/debian trixie/main amd64 groff-base amd64 1.23.0-4 [1180 kB]
Get: 8 http://deb.debian.org/debian trixie/main amd64 bsdextrautils amd64 2.40-8 [92.8 kB]
Get: 9 http://deb.debian.org/debian trixie/main amd64 libpipeline1 amd64 1.5.7-2 [38.0 kB]
Get: 10 http://deb.debian.org/debian trixie/main amd64 man-db amd64 2.12.1-1 [1411 kB]
Get: 11 http://deb.debian.org/debian trixie/main amd64 m4 amd64 1.4.19-4 [287 kB]
Get: 12 http://deb.debian.org/debian trixie/main amd64 autoconf all 2.71-3 [332 kB]
Get: 13 http://deb.debian.org/debian trixie/main amd64 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian trixie/main amd64 automake all 1:1.16.5-1.3 [823 kB]
Get: 15 http://deb.debian.org/debian trixie/main amd64 autopoint all 0.21-14 [496 kB]
Get: 16 http://deb.debian.org/debian trixie/main amd64 libdebhelper-perl all 13.15.3 [88.0 kB]
Get: 17 http://deb.debian.org/debian trixie/main amd64 libtool all 2.4.7-7 [517 kB]
Get: 18 http://deb.debian.org/debian trixie/main amd64 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian trixie/main amd64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian trixie/main amd64 libsub-override-perl all 0.10-1 [10.6 kB]
Get: 21 http://deb.debian.org/debian trixie/main amd64 libfile-stripnondeterminism-perl all 1.13.1-1 [19.4 kB]
Get: 22 http://deb.debian.org/debian trixie/main amd64 dh-strip-nondeterminism all 1.13.1-1 [8620 B]
Get: 23 http://deb.debian.org/debian trixie/main amd64 libelf1t64 amd64 0.191-1+b1 [189 kB]
Get: 24 http://deb.debian.org/debian trixie/main amd64 dwz amd64 0.15-1+b1 [110 kB]
Get: 25 http://deb.debian.org/debian trixie/main amd64 libicu72 amd64 72.1-4+b1 [9395 kB]
Get: 26 http://deb.debian.org/debian trixie/main amd64 libxml2 amd64 2.9.14+dfsg-1.3+b3 [692 kB]
Get: 27 http://deb.debian.org/debian trixie/main amd64 gettext amd64 0.21-14+b1 [1301 kB]
Get: 28 http://deb.debian.org/debian trixie/main amd64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 29 http://deb.debian.org/debian trixie/main amd64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 30 http://deb.debian.org/debian trixie/main amd64 debhelper all 13.15.3 [901 kB]
Get: 31 http://deb.debian.org/debian trixie/main amd64 dh-golang all 1.62 [24.7 kB]
Get: 32 http://deb.debian.org/debian trixie/main amd64 golang-1.22-src all 1.22.2-2 [18.8 MB]
Get: 33 http://deb.debian.org/debian trixie/main amd64 golang-1.22-go amd64 1.22.2-2 [24.7 MB]
Get: 34 http://deb.debian.org/debian trixie/main amd64 golang-src all 2:1.22~3 [5020 B]
Get: 35 http://deb.debian.org/debian trixie/main amd64 golang-go amd64 2:1.22~3 [44.2 kB]
Get: 36 http://deb.debian.org/debian trixie/main amd64 golang-any amd64 2:1.22~3 [5104 B]
Get: 37 http://deb.debian.org/debian trixie/main amd64 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B]
Get: 38 http://deb.debian.org/debian trixie/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 39 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 40 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-sys-dev all 0.19.0-1 [402 kB]
Get: 41 http://deb.debian.org/debian trixie/main amd64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 42 http://deb.debian.org/debian trixie/main amd64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 43 http://deb.debian.org/debian trixie/main amd64 golang-github-fatih-color-dev all 1.16.0-1 [13.8 kB]
Get: 44 http://deb.debian.org/debian trixie/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 45 http://deb.debian.org/debian trixie/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 46 http://deb.debian.org/debian trixie/main amd64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 47 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 48 http://deb.debian.org/debian trixie/main amd64 golang-github-stretchr-testify-dev all 1.8.4-1 [71.8 kB]
Get: 49 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 50 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 51 http://deb.debian.org/debian trixie/main amd64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 52 http://deb.debian.org/debian trixie/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 53 http://deb.debian.org/debian trixie/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 54 http://deb.debian.org/debian trixie/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 55 http://deb.debian.org/debian trixie/main amd64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 56 http://deb.debian.org/debian trixie/main amd64 golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 57 http://deb.debian.org/debian trixie/main amd64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 58 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 59 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 60 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 61 http://deb.debian.org/debian trixie/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 62 http://deb.debian.org/debian trixie/main amd64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 63 http://deb.debian.org/debian trixie/main amd64 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B]
Get: 64 http://deb.debian.org/debian trixie/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 65 http://deb.debian.org/debian trixie/main amd64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 66 http://deb.debian.org/debian trixie/main amd64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 67 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 68 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 69 http://deb.debian.org/debian trixie/main amd64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 70 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-term-dev all 0.19.0-1 [14.7 kB]
Get: 71 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-text-dev all 0.14.0-2 [4001 kB]
Get: 72 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-crypto-dev all 1:0.22.0-1 [1672 kB]
Get: 73 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-net-dev all 1:0.24.0+dfsg-1 [895 kB]
Get: 74 http://deb.debian.org/debian trixie/main amd64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 75 http://deb.debian.org/debian trixie/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 76 http://deb.debian.org/debian trixie/main amd64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 77 http://deb.debian.org/debian trixie/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 78 http://deb.debian.org/debian trixie/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 79 http://deb.debian.org/debian trixie/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 80 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 81 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 82 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-common-dev all 0.53.0-1 [114 kB]
Get: 83 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-sync-dev all 0.6.0-1 [18.5 kB]
Get: 84 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB]
Get: 85 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-client-golang-dev all 1.19.0-1 [179 kB]
Get: 86 http://deb.debian.org/debian trixie/main amd64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 87 http://deb.debian.org/debian trixie/main amd64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 88 http://deb.debian.org/debian trixie/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 89 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 90 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 91 http://deb.debian.org/debian trixie/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-6 [19.5 kB]
Get: 92 http://deb.debian.org/debian trixie/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-6 [56.9 kB]
Get: 93 http://deb.debian.org/debian trixie/main amd64 libssl-dev amd64 3.2.1-3 [2699 kB]
Get: 94 http://deb.debian.org/debian trixie/main amd64 libsasl2-dev amd64 2.1.28+dfsg1-6 [231 kB]
Get: 95 http://deb.debian.org/debian trixie/main amd64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 96 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-mod-dev all 0.17.0-1 [103 kB]
Get: 97 http://deb.debian.org/debian trixie/main amd64 golang-github-yuin-goldmark-dev all 1.7.1-1 [113 kB]
Get: 98 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-tools-dev all 1:0.19.0+ds-1 [1566 kB]
Get: 99 http://deb.debian.org/debian trixie/main amd64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 100 http://deb.debian.org/debian trixie/main amd64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 101 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 102 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 103 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 90.2 MB in 2s (45.9 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19698 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.22_all.deb ...
Unpacking sensible-utils (0.0.22) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3_amd64.deb ...
Unpacking libmagic-mgc (1:5.45-3) ...
Selecting previously unselected package libmagic1t64:amd64.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3_amd64.deb ...
Unpacking libmagic1t64:amd64 (1:5.45-3) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3_amd64.deb ...
Unpacking file (1:5.45-3) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.21-14+b1_amd64.deb ...
Unpacking gettext-base (0.21-14+b1) ...
Selecting previously unselected package libuchardet0:amd64.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b1_amd64.deb ...
Unpacking libuchardet0:amd64 (0.0.8-1+b1) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-4_amd64.deb ...
Unpacking groff-base (1.23.0-4) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40-8_amd64.deb ...
Unpacking bsdextrautils (2.40-8) ...
Selecting previously unselected package libpipeline1:amd64.
Preparing to unpack .../008-libpipeline1_1.5.7-2_amd64.deb ...
Unpacking libpipeline1:amd64 (1.5.7-2) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.12.1-1_amd64.deb ...
Unpacking man-db (2.12.1-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-4_amd64.deb ...
Unpacking m4 (1.4.19-4) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.71-3_all.deb ...
Unpacking autoconf (2.71-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ...
Unpacking automake (1:1.16.5-1.3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.21-14_all.deb ...
Unpacking autopoint (0.21-14) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.15.3_all.deb ...
Unpacking libdebhelper-perl (13.15.3) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.4.7-7_all.deb ...
Unpacking libtool (2.4.7-7) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libsub-override-perl.
Preparing to unpack .../019-libsub-override-perl_0.10-1_all.deb ...
Unpacking libsub-override-perl (0.10-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../020-libfile-stripnondeterminism-perl_1.13.1-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.13.1-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../021-dh-strip-nondeterminism_1.13.1-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.13.1-1) ...
Selecting previously unselected package libelf1t64:amd64.
Preparing to unpack .../022-libelf1t64_0.191-1+b1_amd64.deb ...
Unpacking libelf1t64:amd64 (0.191-1+b1) ...
Selecting previously unselected package dwz.
Preparing to unpack .../023-dwz_0.15-1+b1_amd64.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libicu72:amd64.
Preparing to unpack .../024-libicu72_72.1-4+b1_amd64.deb ...
Unpacking libicu72:amd64 (72.1-4+b1) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../025-libxml2_2.9.14+dfsg-1.3+b3_amd64.deb ...
Unpacking libxml2:amd64 (2.9.14+dfsg-1.3+b3) ...
Selecting previously unselected package gettext.
Preparing to unpack .../026-gettext_0.21-14+b1_amd64.deb ...
Unpacking gettext (0.21-14+b1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../029-debhelper_13.15.3_all.deb ...
Unpacking debhelper (13.15.3) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../030-dh-golang_1.62_all.deb ...
Unpacking dh-golang (1.62) ...
Selecting previously unselected package golang-1.22-src.
Preparing to unpack .../031-golang-1.22-src_1.22.2-2_all.deb ...
Unpacking golang-1.22-src (1.22.2-2) ...
Selecting previously unselected package golang-1.22-go.
Preparing to unpack .../032-golang-1.22-go_1.22.2-2_amd64.deb ...
Unpacking golang-1.22-go (1.22.2-2) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../033-golang-src_2%3a1.22~3_all.deb ...
Unpacking golang-src (2:1.22~3) ...
Selecting previously unselected package golang-go:amd64.
Preparing to unpack .../034-golang-go_2%3a1.22~3_amd64.deb ...
Unpacking golang-go:amd64 (2:1.22~3) ...
Selecting previously unselected package golang-any:amd64.
Preparing to unpack .../035-golang-any_2%3a1.22~3_amd64.deb ...
Unpacking golang-any:amd64 (2:1.22~3) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../039-golang-golang-x-sys-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../042-golang-github-fatih-color-dev_1.16.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.16.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.8.4-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.8.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../055-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../066-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../067-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../068-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../069-golang-golang-x-term-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.19.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../070-golang-golang-x-text-dev_0.14.0-2_all.deb ...
Unpacking golang-golang-x-text-dev (0.14.0-2) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../071-golang-golang-x-crypto-dev_1%3a0.22.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.22.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../072-golang-golang-x-net-dev_1%3a0.24.0+dfsg-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.24.0+dfsg-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../073-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../074-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../075-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../076-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../077-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../078-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../079-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../080-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../081-golang-github-prometheus-common-dev_0.53.0-1_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.53.0-1) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../082-golang-golang-x-sync-dev_0.6.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.6.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../083-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../084-golang-github-prometheus-client-golang-dev_1.19.0-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.19.0-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../085-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../086-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../087-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../088-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../089-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../090-libsasl2-modules-db_2.1.28+dfsg1-6_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-6) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../091-libsasl2-2_2.1.28+dfsg1-6_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-6) ...
Selecting previously unselected package libssl-dev:amd64.
Preparing to unpack .../092-libssl-dev_3.2.1-3_amd64.deb ...
Unpacking libssl-dev:amd64 (3.2.1-3) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../093-libsasl2-dev_2.1.28+dfsg1-6_amd64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-6) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../094-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../095-golang-golang-x-mod-dev_0.17.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.17.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../096-golang-github-yuin-goldmark-dev_1.7.1-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.1-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../097-golang-golang-x-tools-dev_1%3a0.19.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.19.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../098-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../099-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../100-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../101-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../102-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up libpipeline1:amd64 (1.5.7-2) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:amd64 (72.1-4+b1) ...
Setting up bsdextrautils (2.40-8) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.15.3) ...
Setting up golang-golang-x-sys-dev (0.19.0-1) ...
Setting up libmagic1t64:amd64 (1:5.45-3) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.21-14+b1) ...
Setting up m4 (1.4.19-4) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.19.0-1) ...
Setting up file (1:5.45-3) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libelf1t64:amd64 (0.191-1+b1) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-6) ...
Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.8.4-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.1-1) ...
Setting up golang-1.22-src (1.22.2-2) ...
Setting up libssl-dev:amd64 (3.2.1-3) ...
Setting up golang-golang-x-sync-dev (0.6.0-1) ...
Setting up golang-golang-x-mod-dev (0.17.0-1) ...
Setting up autopoint (0.21-14) ...
Setting up libsasl2-2:amd64 (2.1.28+dfsg1-6) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.71-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up sensible-utils (0.0.22) ...
Setting up golang-golang-x-text-dev (0.14.0-2) ...
Setting up libuchardet0:amd64 (0.0.8-1+b1) ...
Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up libsub-override-perl (0.10-1) ...
Setting up golang-golang-x-crypto-dev (1:0.22.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.22~3) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:amd64 (2.9.14+dfsg-1.3+b3) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.16.5-1.3) ...
update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.13.1-1) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.21-14+b1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.4.7-7) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-1.22-go (1.22.2-2) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg1-6) ...
Setting up golang-golang-x-net-dev (1:0.24.0+dfsg-1) ...
Setting up dh-strip-nondeterminism (1.13.1-1) ...
Setting up groff-base (1.23.0-4) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.16.0-1) ...
Setting up golang-go:amd64 (2:1.22~3) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.19.0+ds-1) ...
Setting up man-db (2.12.1-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-any:amd64 (2:1.22~3) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.15.3) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.62) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-common-dev (0.53.0-1) ...
Setting up golang-github-prometheus-client-golang-dev (1.19.0-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.38-7) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: user script /srv/workspace/pbuilder/2764266/tmp/hooks/A99_set_merged_usr starting
Not re-configuring usrmerge for trixie
I: user script /srv/workspace/pbuilder/2764266/tmp/hooks/A99_set_merged_usr finished
hostname: Name or service not known
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-x86_64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/godebugs
internal/goos
internal/goexperiment
internal/race
internal/coverage/rtcov
unicode
internal/unsafeheader
internal/goarch
unicode/utf8
runtime/internal/math
runtime/internal/sys
container/list
internal/itoa
math/bits
crypto/internal/alias
sync/atomic
crypto/subtle
runtime/internal/syscall
runtime/internal/atomic
internal/abi
cmp
encoding
internal/chacha8rand
unicode/utf16
crypto/internal/boring/sig
internal/cpu
log/internal
slices
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
internal/bytealg
math
runtime
sync
internal/reflectlite
internal/testlog
internal/bisect
internal/singleflight
internal/godebug
runtime/cgo
internal/intern
math/rand
errors
sort
internal/oserror
internal/safefilepath
path
github.com/hashicorp/golang-lru/simplelru
strconv
vendor/golang.org/x/net/dns/dnsmessage
io
crypto/internal/nistec/fiat
syscall
hash
crypto/internal/randutil
strings
bytes
internal/saferio
hash/crc32
hash/crc64
vendor/golang.org/x/text/transform
bufio
crypto
reflect
time
regexp/syntax
github.com/hashicorp/go-immutable-radix
crypto/rc4
html
internal/syscall/unix
internal/syscall/execenv
net/netip
net/http/internal/ascii
regexp
io/fs
context
internal/poll
embed
os
os/signal
io/ioutil
internal/sysinfo
path/filepath
vendor/golang.org/x/sys/cpu
internal/fmtsort
encoding/binary
fmt
crypto/cipher
encoding/base64
vendor/golang.org/x/crypto/internal/poly1305
crypto/internal/nistec
crypto/internal/edwards25519/field
golang.org/x/sys/unix
crypto/md5
crypto/internal/boring
crypto/des
vendor/golang.org/x/crypto/chacha20
crypto/sha512
crypto/aes
crypto/hmac
crypto/sha1
crypto/sha256
encoding/pem
log
math/big
encoding/json
compress/flate
crypto/internal/edwards25519
encoding/hex
net/url
vendor/golang.org/x/crypto/chacha20poly1305
crypto/ecdh
vendor/golang.org/x/crypto/hkdf
vendor/golang.org/x/text/unicode/bidi
vendor/golang.org/x/net/http2/hpack
vendor/golang.org/x/text/unicode/norm
mime
mime/quotedprintable
compress/gzip
net/http/internal
encoding/gob
go/token
flag
text/template/parse
runtime/debug
runtime/trace
vendor/golang.org/x/text/secure/bidirule
testing
text/template
vendor/golang.org/x/net/idna
crypto/rand
crypto/elliptic
crypto/dsa
crypto/internal/boring/bbig
encoding/asn1
crypto/internal/bigmod
net
crypto/ed25519
crypto/rsa
html/template
github.com/mattn/go-isatty
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
github.com/mattn/go-colorable
github.com/fatih/color
crypto/ecdsa
github.com/hashicorp/go-hclog
vendor/golang.org/x/net/http/httpproxy
net/textproto
crypto/x509
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
github.com/armon/go-metrics
net/rpc
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-x86_64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
?   	github.com/hashicorp/raft/bench	[no test files]
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2024-05-07T18:34:22.400Z [INFO]  creating new snapshot: path=/tmp/raft549191828/raft129989491/snapshots/3-10-1715106862400.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.18s)
=== RUN   TestFileSS_CreateSnapshot
2024-05-07T18:34:22.583Z [INFO]  creating new snapshot: path=/tmp/raft1635360958/snapshots/3-10-1715106862583.tmp
2024-05-07T18:34:22.878Z [WARN]  found temporary snapshot: name=3-10-1715106862583.tmp
--- PASS: TestFileSS_CreateSnapshot (0.50s)
=== RUN   TestFileSS_CancelSnapshot
2024-05-07T18:34:23.084Z [INFO]  creating new snapshot: path=/tmp/raft2854174630/snapshots/3-10-1715106863084.tmp
--- PASS: TestFileSS_CancelSnapshot (0.01s)
=== RUN   TestFileSS_Retention
2024-05-07T18:34:23.096Z [INFO]  creating new snapshot: path=/tmp/raft1266111654/snapshots/3-10-1715106863096.tmp
2024-05-07T18:34:23.118Z [INFO]  creating new snapshot: path=/tmp/raft1266111654/snapshots/3-11-1715106863118.tmp
2024-05-07T18:34:23.142Z [INFO]  creating new snapshot: path=/tmp/raft1266111654/snapshots/3-12-1715106863142.tmp
2024-05-07T18:34:23.162Z [INFO]  reaping snapshot: path=/tmp/raft1266111654/snapshots/3-10-1715106863096
2024-05-07T18:34:23.162Z [INFO]  creating new snapshot: path=/tmp/raft1266111654/snapshots/3-13-1715106863162.tmp
2024-05-07T18:34:23.183Z [INFO]  reaping snapshot: path=/tmp/raft1266111654/snapshots/3-11-1715106863118
2024-05-07T18:34:23.183Z [INFO]  creating new snapshot: path=/tmp/raft1266111654/snapshots/3-14-1715106863183.tmp
2024-05-07T18:34:23.206Z [INFO]  reaping snapshot: path=/tmp/raft1266111654/snapshots/3-12-1715106863142
--- PASS: TestFileSS_Retention (0.11s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2024-05-07T18:34:23.207Z [INFO]  creating new snapshot: path=/tmp/raft1579927454/snapshots/5-130350-1715106863207.tmp
2024-05-07T18:34:23.230Z [INFO]  creating new snapshot: path=/tmp/raft1579927454/snapshots/36-204917-1715106863230.tmp
--- PASS: TestFileSS_Ordering (0.04s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.04s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.03s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2024-05-07T18:34:23.403Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2024-05-07T18:34:23.455Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.17s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.07s)
=== RUN   TestNetworkTransport_RequestVote
2024-05-07T18:34:23.522Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.02s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:23.543Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:23.550Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:23.562Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:23.586Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:23.630Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:23.714Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:23.878Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:24.202Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:24.846Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:25.850Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-05-07T18:34:26.854Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2024-05-07T18:34:27.547Z [INFO]  server-531d32db-c420-9de2-cfe7-cb9c5d2cd6a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-531d32db-c420-9de2-cfe7-cb9c5d2cd6a6 Address:531d32db-c420-9de2-cfe7-cb9c5d2cd6a6}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2024-05-07T18:34:27.548Z [INFO]  server-fd253328-2438-f792-d879-9a6b58734702: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd253328-2438-f792-d879-9a6b58734702 Address:fd253328-2438-f792-d879-9a6b58734702}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2024-05-07T18:34:27.548Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: initial configuration: index=0 servers=[]
2024-05-07T18:34:27.548Z [INFO]  server-bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387: initial configuration: index=0 servers=[]
2024-05-07T18:34:27.548Z [INFO]  server-dc2e21cf-a0cc-c41b-0639-23e9c9739027: initial configuration: index=0 servers=[]
2024-05-07T18:34:27.548Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: entering follower state: follower="Node at e15a3ac9-2f41-ec6e-459b-26ef18c51f8f [Follower]" leader-address= leader-id=
2024-05-07T18:34:27.548Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:27.548Z [INFO]  server-bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387: entering follower state: follower="Node at bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387 [Follower]" leader-address= leader-id=
2024-05-07T18:34:27.548Z [INFO]  server-dc2e21cf-a0cc-c41b-0639-23e9c9739027: entering follower state: follower="Node at dc2e21cf-a0cc-c41b-0639-23e9c9739027 [Follower]" leader-address= leader-id=
2024-05-07T18:34:27.614Z [WARN]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:27.614Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: entering candidate state: node="Node at e15a3ac9-2f41-ec6e-459b-26ef18c51f8f [Candidate]" term=2
2024-05-07T18:34:27.614Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: election won: term=2 tally=2
2024-05-07T18:34:27.614Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: entering leader state: leader="Node at e15a3ac9-2f41-ec6e-459b-26ef18c51f8f [Leader]"
2024-05-07T18:34:27.614Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: added peer, starting replication: peer=server-bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387
2024-05-07T18:34:27.614Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: added peer, starting replication: peer=server-dc2e21cf-a0cc-c41b-0639-23e9c9739027
2024-05-07T18:34:27.614Z [WARN]  server-dc2e21cf-a0cc-c41b-0639-23e9c9739027: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2024-05-07T18:34:27.614Z [WARN]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: appendEntries rejected, sending older logs: peer="{Voter server-dc2e21cf-a0cc-c41b-0639-23e9c9739027 dc2e21cf-a0cc-c41b-0639-23e9c9739027}" next=1
2024-05-07T18:34:27.614Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: pipelining replication: peer="{Voter server-dc2e21cf-a0cc-c41b-0639-23e9c9739027 dc2e21cf-a0cc-c41b-0639-23e9c9739027}"
2024-05-07T18:34:27.614Z [WARN]  server-bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2024-05-07T18:34:27.614Z [WARN]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: appendEntries rejected, sending older logs: peer="{Voter server-bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387 bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387}" next=1
2024-05-07T18:34:27.614Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: pipelining replication: peer="{Voter server-bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387 bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387}"
2024-05-07T18:34:27.722Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:27.614914332 +0000 UTC m=+5.216032572 (2 nodes), 66.297511ms from start of poll, 66.446447ms from cluster start. Timeout at 2024-05-07 18:34:27.722488323 +0000 UTC m=+5.323606563, 107.573991ms after stability
2024-05-07T18:34:27.722Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:27.830Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:27.722602385 +0000 UTC m=+5.323720625 (1 nodes), 2.615µs from start of poll, 174.1345ms from cluster start. Timeout at 2024-05-07 18:34:27.830492605 +0000 UTC m=+5.431610845, 107.89022ms after stability
2024-05-07T18:34:27.846Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: aborting pipeline replication: peer="{Voter server-bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387 bb88a194-2e74-b9ee-e8dc-cd8ebf1f0387}"
2024-05-07T18:34:27.846Z [INFO]  server-e15a3ac9-2f41-ec6e-459b-26ef18c51f8f: aborting pipeline replication: peer="{Voter server-dc2e21cf-a0cc-c41b-0639-23e9c9739027 dc2e21cf-a0cc-c41b-0639-23e9c9739027}"
--- PASS: TestRaft_LiveBootstrap (0.30s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2024-05-07T18:34:27.847Z [INFO]  server-42364a3e-9e2f-54e1-0374-335edd6309bb: initial configuration: index=0 servers=[]
2024-05-07T18:34:27.847Z [INFO]  server-52fbca31-4d3a-528b-7320-8c055fb6341e: initial configuration: index=0 servers=[]
2024-05-07T18:34:27.847Z [INFO]  server-42364a3e-9e2f-54e1-0374-335edd6309bb: entering follower state: follower="Node at 42364a3e-9e2f-54e1-0374-335edd6309bb [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2024-05-07T18:34:27.847Z [INFO]  server-3e7b2594-70e7-d35c-a966-2bafcd9b7ed3: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2024-05-07T18:34:27.848Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7823067-c465-680f-76ce-09b7c0506ff0 Address:d7823067-c465-680f-76ce-09b7c0506ff0} {Suffrage:Voter ID:server-e832daeb-b574-25b0-a22b-9f7570c31e8d Address:e832daeb-b574-25b0-a22b-9f7570c31e8d} {Suffrage:Voter ID:server-ee974dcb-c713-1007-fced-01dd630b70c4 Address:ee974dcb-c713-1007-fced-01dd630b70c4}]"
2024-05-07T18:34:27.848Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7823067-c465-680f-76ce-09b7c0506ff0 Address:d7823067-c465-680f-76ce-09b7c0506ff0} {Suffrage:Voter ID:server-e832daeb-b574-25b0-a22b-9f7570c31e8d Address:e832daeb-b574-25b0-a22b-9f7570c31e8d} {Suffrage:Voter ID:server-ee974dcb-c713-1007-fced-01dd630b70c4 Address:ee974dcb-c713-1007-fced-01dd630b70c4}]"
2024-05-07T18:34:27.848Z [INFO]  server-ee974dcb-c713-1007-fced-01dd630b70c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7823067-c465-680f-76ce-09b7c0506ff0 Address:d7823067-c465-680f-76ce-09b7c0506ff0} {Suffrage:Voter ID:server-e832daeb-b574-25b0-a22b-9f7570c31e8d Address:e832daeb-b574-25b0-a22b-9f7570c31e8d} {Suffrage:Voter ID:server-ee974dcb-c713-1007-fced-01dd630b70c4 Address:ee974dcb-c713-1007-fced-01dd630b70c4}]"
2024-05-07T18:34:27.848Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:27.848Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: entering follower state: follower="Node at d7823067-c465-680f-76ce-09b7c0506ff0 [Follower]" leader-address= leader-id=
2024-05-07T18:34:27.848Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: entering follower state: follower="Node at e832daeb-b574-25b0-a22b-9f7570c31e8d [Follower]" leader-address= leader-id=
2024-05-07T18:34:27.848Z [INFO]  server-ee974dcb-c713-1007-fced-01dd630b70c4: entering follower state: follower="Node at ee974dcb-c713-1007-fced-01dd630b70c4 [Follower]" leader-address= leader-id=
2024-05-07T18:34:27.902Z [WARN]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:27.902Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: entering candidate state: node="Node at e832daeb-b574-25b0-a22b-9f7570c31e8d [Candidate]" term=2
2024-05-07T18:34:27.902Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: election won: term=2 tally=2
2024-05-07T18:34:27.902Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: entering leader state: leader="Node at e832daeb-b574-25b0-a22b-9f7570c31e8d [Leader]"
2024-05-07T18:34:27.902Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: added peer, starting replication: peer=server-d7823067-c465-680f-76ce-09b7c0506ff0
2024-05-07T18:34:27.902Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: added peer, starting replication: peer=server-ee974dcb-c713-1007-fced-01dd630b70c4
2024-05-07T18:34:27.902Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: pipelining replication: peer="{Voter server-ee974dcb-c713-1007-fced-01dd630b70c4 ee974dcb-c713-1007-fced-01dd630b70c4}"
2024-05-07T18:34:27.902Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: pipelining replication: peer="{Voter server-d7823067-c465-680f-76ce-09b7c0506ff0 d7823067-c465-680f-76ce-09b7c0506ff0}"
2024-05-07T18:34:28.010Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:27.90293804 +0000 UTC m=+5.504056280 (1 nodes), 54.369445ms from start of poll, 54.640338ms from cluster start. Timeout at 2024-05-07 18:34:28.010485633 +0000 UTC m=+5.611603873, 107.547593ms after stability
2024-05-07T18:34:28.010Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: aborting pipeline replication: peer="{Voter server-ee974dcb-c713-1007-fced-01dd630b70c4 ee974dcb-c713-1007-fced-01dd630b70c4}"
2024-05-07T18:34:28.010Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: aborting pipeline replication: peer="{Voter server-d7823067-c465-680f-76ce-09b7c0506ff0 d7823067-c465-680f-76ce-09b7c0506ff0}"
2024-05-07T18:34:28.010Z [INFO]  creating new snapshot: path=/tmp/raft98785355/snapshots/2-2-1715106868010.tmp
2024-05-07T18:34:28.011Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: starting restore from snapshot: id=2-2-1715106868010 last-index=2 last-term=2 size-in-bytes=1
2024-05-07T18:34:28.011Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: snapshot restore progress: id=2-2-1715106868010 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2024-05-07T18:34:28.011Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: restored from snapshot: id=2-2-1715106868010 last-index=2 last-term=2 size-in-bytes=1
2024-05-07T18:34:28.011Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7823067-c465-680f-76ce-09b7c0506ff0 Address:d7823067-c465-680f-76ce-09b7c0506ff0} {Suffrage:Voter ID:server-e832daeb-b574-25b0-a22b-9f7570c31e8d Address:e832daeb-b574-25b0-a22b-9f7570c31e8d} {Suffrage:Voter ID:server-ee974dcb-c713-1007-fced-01dd630b70c4 Address:ee974dcb-c713-1007-fced-01dd630b70c4}]"
2024-05-07T18:34:28.011Z [INFO]  creating new snapshot: path=/tmp/raft1895207847/snapshots/2-2-1715106868011.tmp
2024-05-07T18:34:28.011Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: starting restore from snapshot: id=2-2-1715106868011 last-index=2 last-term=2 size-in-bytes=1
2024-05-07T18:34:28.011Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: snapshot restore progress: id=2-2-1715106868011 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2024-05-07T18:34:28.011Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: restored from snapshot: id=2-2-1715106868011 last-index=2 last-term=2 size-in-bytes=1
2024-05-07T18:34:28.011Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7823067-c465-680f-76ce-09b7c0506ff0 Address:d7823067-c465-680f-76ce-09b7c0506ff0} {Suffrage:Voter ID:server-e832daeb-b574-25b0-a22b-9f7570c31e8d Address:e832daeb-b574-25b0-a22b-9f7570c31e8d} {Suffrage:Voter ID:server-ee974dcb-c713-1007-fced-01dd630b70c4 Address:ee974dcb-c713-1007-fced-01dd630b70c4}]"
2024-05-07T18:34:28.011Z [INFO]  creating new snapshot: path=/tmp/raft3476777211/snapshots/2-2-1715106868011.tmp
2024-05-07T18:34:28.012Z [INFO]  server-ee974dcb-c713-1007-fced-01dd630b70c4: starting restore from snapshot: id=2-2-1715106868011 last-index=2 last-term=2 size-in-bytes=1
2024-05-07T18:34:28.012Z [INFO]  server-ee974dcb-c713-1007-fced-01dd630b70c4: snapshot restore progress: id=2-2-1715106868011 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2024-05-07T18:34:28.012Z [INFO]  server-ee974dcb-c713-1007-fced-01dd630b70c4: restored from snapshot: id=2-2-1715106868011 last-index=2 last-term=2 size-in-bytes=1
2024-05-07T18:34:28.012Z [INFO]  server-ee974dcb-c713-1007-fced-01dd630b70c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7823067-c465-680f-76ce-09b7c0506ff0 Address:d7823067-c465-680f-76ce-09b7c0506ff0} {Suffrage:Voter ID:server-e832daeb-b574-25b0-a22b-9f7570c31e8d Address:e832daeb-b574-25b0-a22b-9f7570c31e8d} {Suffrage:Voter ID:server-ee974dcb-c713-1007-fced-01dd630b70c4 Address:ee974dcb-c713-1007-fced-01dd630b70c4}]"
2024-05-07T18:34:28.012Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: entering follower state: follower="Node at d7823067-c465-680f-76ce-09b7c0506ff0 [Follower]" leader-address= leader-id=
2024-05-07T18:34:28.012Z [INFO]  server-e832daeb-b574-25b0-a22b-9f7570c31e8d: entering follower state: follower="Node at e832daeb-b574-25b0-a22b-9f7570c31e8d [Follower]" leader-address= leader-id=
2024-05-07T18:34:28.012Z [INFO]  server-ee974dcb-c713-1007-fced-01dd630b70c4: entering follower state: follower="Node at ee974dcb-c713-1007-fced-01dd630b70c4 [Follower]" leader-address= leader-id=
2024-05-07T18:34:28.093Z [WARN]  server-d7823067-c465-680f-76ce-09b7c0506ff0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:28.093Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: entering candidate state: node="Node at d7823067-c465-680f-76ce-09b7c0506ff0 [Candidate]" term=3
2024-05-07T18:34:28.093Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: election won: term=3 tally=2
2024-05-07T18:34:28.093Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: entering leader state: leader="Node at d7823067-c465-680f-76ce-09b7c0506ff0 [Leader]"
2024-05-07T18:34:28.093Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: added peer, starting replication: peer=server-e832daeb-b574-25b0-a22b-9f7570c31e8d
2024-05-07T18:34:28.093Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: added peer, starting replication: peer=server-ee974dcb-c713-1007-fced-01dd630b70c4
2024-05-07T18:34:28.093Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: pipelining replication: peer="{Voter server-ee974dcb-c713-1007-fced-01dd630b70c4 ee974dcb-c713-1007-fced-01dd630b70c4}"
2024-05-07T18:34:28.093Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: pipelining replication: peer="{Voter server-e832daeb-b574-25b0-a22b-9f7570c31e8d e832daeb-b574-25b0-a22b-9f7570c31e8d}"
2024-05-07T18:34:28.334Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:28.442Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:28.334604225 +0000 UTC m=+5.935722465 (1 nodes), 19.867µs from start of poll, 486.306523ms from cluster start. Timeout at 2024-05-07 18:34:28.442484016 +0000 UTC m=+6.043602256, 107.879791ms after stability
2024-05-07T18:34:28.442Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: aborting pipeline replication: peer="{Voter server-ee974dcb-c713-1007-fced-01dd630b70c4 ee974dcb-c713-1007-fced-01dd630b70c4}"
2024-05-07T18:34:28.442Z [INFO]  server-d7823067-c465-680f-76ce-09b7c0506ff0: aborting pipeline replication: peer="{Voter server-e832daeb-b574-25b0-a22b-9f7570c31e8d e832daeb-b574-25b0-a22b-9f7570c31e8d}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2024-05-07T18:34:28.443Z [INFO]  server-36c0e71f-25aa-64a2-393e-3336cbfd16b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36c0e71f-25aa-64a2-393e-3336cbfd16b3 Address:36c0e71f-25aa-64a2-393e-3336cbfd16b3} {Suffrage:Voter ID:server-9e432013-d40e-0fb3-670a-ef7bad392cd1 Address:9e432013-d40e-0fb3-670a-ef7bad392cd1} {Suffrage:Voter ID:server-24a3754d-c814-8598-01aa-f1a7f60f49cb Address:24a3754d-c814-8598-01aa-f1a7f60f49cb}]"
2024-05-07T18:34:28.443Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36c0e71f-25aa-64a2-393e-3336cbfd16b3 Address:36c0e71f-25aa-64a2-393e-3336cbfd16b3} {Suffrage:Voter ID:server-9e432013-d40e-0fb3-670a-ef7bad392cd1 Address:9e432013-d40e-0fb3-670a-ef7bad392cd1} {Suffrage:Voter ID:server-24a3754d-c814-8598-01aa-f1a7f60f49cb Address:24a3754d-c814-8598-01aa-f1a7f60f49cb}]"
2024-05-07T18:34:28.443Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36c0e71f-25aa-64a2-393e-3336cbfd16b3 Address:36c0e71f-25aa-64a2-393e-3336cbfd16b3} {Suffrage:Voter ID:server-9e432013-d40e-0fb3-670a-ef7bad392cd1 Address:9e432013-d40e-0fb3-670a-ef7bad392cd1} {Suffrage:Voter ID:server-24a3754d-c814-8598-01aa-f1a7f60f49cb Address:24a3754d-c814-8598-01aa-f1a7f60f49cb}]"
2024-05-07T18:34:28.443Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:28.443Z [INFO]  server-36c0e71f-25aa-64a2-393e-3336cbfd16b3: entering follower state: follower="Node at 36c0e71f-25aa-64a2-393e-3336cbfd16b3 [Follower]" leader-address= leader-id=
2024-05-07T18:34:28.443Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: entering follower state: follower="Node at 9e432013-d40e-0fb3-670a-ef7bad392cd1 [Follower]" leader-address= leader-id=
2024-05-07T18:34:28.443Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: entering follower state: follower="Node at 24a3754d-c814-8598-01aa-f1a7f60f49cb [Follower]" leader-address= leader-id=
2024-05-07T18:34:28.510Z [WARN]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:28.510Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: entering candidate state: node="Node at 24a3754d-c814-8598-01aa-f1a7f60f49cb [Candidate]" term=2
2024-05-07T18:34:28.510Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: election won: term=2 tally=2
2024-05-07T18:34:28.510Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: entering leader state: leader="Node at 24a3754d-c814-8598-01aa-f1a7f60f49cb [Leader]"
2024-05-07T18:34:28.510Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: added peer, starting replication: peer=server-36c0e71f-25aa-64a2-393e-3336cbfd16b3
2024-05-07T18:34:28.510Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: added peer, starting replication: peer=server-9e432013-d40e-0fb3-670a-ef7bad392cd1
2024-05-07T18:34:28.510Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: pipelining replication: peer="{Voter server-9e432013-d40e-0fb3-670a-ef7bad392cd1 9e432013-d40e-0fb3-670a-ef7bad392cd1}"
2024-05-07T18:34:28.510Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: pipelining replication: peer="{Voter server-36c0e71f-25aa-64a2-393e-3336cbfd16b3 36c0e71f-25aa-64a2-393e-3336cbfd16b3}"
2024-05-07T18:34:28.618Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:28.510898022 +0000 UTC m=+6.112016262 (1 nodes), 67.046383ms from start of poll, 67.26583ms from cluster start. Timeout at 2024-05-07 18:34:28.618490519 +0000 UTC m=+6.219608759, 107.592497ms after stability
2024-05-07T18:34:28.618Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: aborting pipeline replication: peer="{Voter server-36c0e71f-25aa-64a2-393e-3336cbfd16b3 36c0e71f-25aa-64a2-393e-3336cbfd16b3}"
2024-05-07T18:34:28.619Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: aborting pipeline replication: peer="{Voter server-9e432013-d40e-0fb3-670a-ef7bad392cd1 9e432013-d40e-0fb3-670a-ef7bad392cd1}"
2024-05-07T18:34:28.619Z [INFO]  creating new snapshot: path=/tmp/raft1087761315/snapshots/2-6-1715106868619.tmp
2024-05-07T18:34:28.619Z [INFO]  server-36c0e71f-25aa-64a2-393e-3336cbfd16b3: starting restore from snapshot: id=2-6-1715106868619 last-index=6 last-term=2 size-in-bytes=25
2024-05-07T18:34:28.619Z [INFO]  server-36c0e71f-25aa-64a2-393e-3336cbfd16b3: snapshot restore progress: id=2-6-1715106868619 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2024-05-07T18:34:28.619Z [INFO]  server-36c0e71f-25aa-64a2-393e-3336cbfd16b3: restored from snapshot: id=2-6-1715106868619 last-index=6 last-term=2 size-in-bytes=25
2024-05-07T18:34:28.619Z [INFO]  server-36c0e71f-25aa-64a2-393e-3336cbfd16b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36c0e71f-25aa-64a2-393e-3336cbfd16b3 Address:36c0e71f-25aa-64a2-393e-3336cbfd16b3} {Suffrage:Voter ID:server-9e432013-d40e-0fb3-670a-ef7bad392cd1 Address:9e432013-d40e-0fb3-670a-ef7bad392cd1} {Suffrage:Voter ID:server-24a3754d-c814-8598-01aa-f1a7f60f49cb Address:24a3754d-c814-8598-01aa-f1a7f60f49cb}]"
2024-05-07T18:34:28.619Z [INFO]  creating new snapshot: path=/tmp/raft1890580107/snapshots/2-6-1715106868619.tmp
2024-05-07T18:34:28.620Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: starting restore from snapshot: id=2-6-1715106868619 last-index=6 last-term=2 size-in-bytes=25
2024-05-07T18:34:28.620Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: snapshot restore progress: id=2-6-1715106868619 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2024-05-07T18:34:28.620Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: restored from snapshot: id=2-6-1715106868619 last-index=6 last-term=2 size-in-bytes=25
2024-05-07T18:34:28.620Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36c0e71f-25aa-64a2-393e-3336cbfd16b3 Address:36c0e71f-25aa-64a2-393e-3336cbfd16b3} {Suffrage:Voter ID:server-9e432013-d40e-0fb3-670a-ef7bad392cd1 Address:9e432013-d40e-0fb3-670a-ef7bad392cd1} {Suffrage:Voter ID:server-24a3754d-c814-8598-01aa-f1a7f60f49cb Address:24a3754d-c814-8598-01aa-f1a7f60f49cb}]"
2024-05-07T18:34:28.620Z [INFO]  creating new snapshot: path=/tmp/raft964437609/snapshots/2-6-1715106868620.tmp
2024-05-07T18:34:28.620Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: starting restore from snapshot: id=2-6-1715106868620 last-index=6 last-term=2 size-in-bytes=25
2024-05-07T18:34:28.620Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: snapshot restore progress: id=2-6-1715106868620 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2024-05-07T18:34:28.620Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: restored from snapshot: id=2-6-1715106868620 last-index=6 last-term=2 size-in-bytes=25
2024-05-07T18:34:28.620Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36c0e71f-25aa-64a2-393e-3336cbfd16b3 Address:36c0e71f-25aa-64a2-393e-3336cbfd16b3} {Suffrage:Voter ID:server-9e432013-d40e-0fb3-670a-ef7bad392cd1 Address:9e432013-d40e-0fb3-670a-ef7bad392cd1} {Suffrage:Voter ID:server-24a3754d-c814-8598-01aa-f1a7f60f49cb Address:24a3754d-c814-8598-01aa-f1a7f60f49cb}]"
2024-05-07T18:34:28.620Z [INFO]  server-36c0e71f-25aa-64a2-393e-3336cbfd16b3: entering follower state: follower="Node at 36c0e71f-25aa-64a2-393e-3336cbfd16b3 [Follower]" leader-address= leader-id=
2024-05-07T18:34:28.620Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: entering follower state: follower="Node at 9e432013-d40e-0fb3-670a-ef7bad392cd1 [Follower]" leader-address= leader-id=
2024-05-07T18:34:28.620Z [INFO]  server-24a3754d-c814-8598-01aa-f1a7f60f49cb: entering follower state: follower="Node at 24a3754d-c814-8598-01aa-f1a7f60f49cb [Follower]" leader-address= leader-id=
2024-05-07T18:34:28.694Z [WARN]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:28.694Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: entering candidate state: node="Node at 9e432013-d40e-0fb3-670a-ef7bad392cd1 [Candidate]" term=3
2024-05-07T18:34:28.694Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: election won: term=3 tally=2
2024-05-07T18:34:28.694Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: entering leader state: leader="Node at 9e432013-d40e-0fb3-670a-ef7bad392cd1 [Leader]"
2024-05-07T18:34:28.694Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: added peer, starting replication: peer=server-36c0e71f-25aa-64a2-393e-3336cbfd16b3
2024-05-07T18:34:28.694Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: added peer, starting replication: peer=server-24a3754d-c814-8598-01aa-f1a7f60f49cb
2024-05-07T18:34:28.694Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: pipelining replication: peer="{Voter server-24a3754d-c814-8598-01aa-f1a7f60f49cb 24a3754d-c814-8598-01aa-f1a7f60f49cb}"
2024-05-07T18:34:28.694Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: pipelining replication: peer="{Voter server-36c0e71f-25aa-64a2-393e-3336cbfd16b3 36c0e71f-25aa-64a2-393e-3336cbfd16b3}"
2024-05-07T18:34:28.938Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:29.046Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:28.938605374 +0000 UTC m=+6.539723624 (1 nodes), 24.134µs from start of poll, 494.973192ms from cluster start. Timeout at 2024-05-07 18:34:29.046494674 +0000 UTC m=+6.647612924, 107.8893ms after stability
2024-05-07T18:34:29.046Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: aborting pipeline replication: peer="{Voter server-24a3754d-c814-8598-01aa-f1a7f60f49cb 24a3754d-c814-8598-01aa-f1a7f60f49cb}"
2024-05-07T18:34:29.046Z [INFO]  server-9e432013-d40e-0fb3-670a-ef7bad392cd1: aborting pipeline replication: peer="{Voter server-36c0e71f-25aa-64a2-393e-3336cbfd16b3 36c0e71f-25aa-64a2-393e-3336cbfd16b3}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2024-05-07T18:34:29.047Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a86ff64-da9e-b194-198e-62c3ccff74da Address:3a86ff64-da9e-b194-198e-62c3ccff74da} {Suffrage:Voter ID:server-ff8010cf-fe92-2260-1dea-910d58a048c8 Address:ff8010cf-fe92-2260-1dea-910d58a048c8} {Suffrage:Voter ID:server-10cb60c1-0099-a057-9905-5cbc25043093 Address:10cb60c1-0099-a057-9905-5cbc25043093}]"
2024-05-07T18:34:29.048Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a86ff64-da9e-b194-198e-62c3ccff74da Address:3a86ff64-da9e-b194-198e-62c3ccff74da} {Suffrage:Voter ID:server-ff8010cf-fe92-2260-1dea-910d58a048c8 Address:ff8010cf-fe92-2260-1dea-910d58a048c8} {Suffrage:Voter ID:server-10cb60c1-0099-a057-9905-5cbc25043093 Address:10cb60c1-0099-a057-9905-5cbc25043093}]"
2024-05-07T18:34:29.048Z [INFO]  server-10cb60c1-0099-a057-9905-5cbc25043093: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a86ff64-da9e-b194-198e-62c3ccff74da Address:3a86ff64-da9e-b194-198e-62c3ccff74da} {Suffrage:Voter ID:server-ff8010cf-fe92-2260-1dea-910d58a048c8 Address:ff8010cf-fe92-2260-1dea-910d58a048c8} {Suffrage:Voter ID:server-10cb60c1-0099-a057-9905-5cbc25043093 Address:10cb60c1-0099-a057-9905-5cbc25043093}]"
2024-05-07T18:34:29.048Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:29.048Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: entering follower state: follower="Node at 3a86ff64-da9e-b194-198e-62c3ccff74da [Follower]" leader-address= leader-id=
2024-05-07T18:34:29.048Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: entering follower state: follower="Node at ff8010cf-fe92-2260-1dea-910d58a048c8 [Follower]" leader-address= leader-id=
2024-05-07T18:34:29.048Z [INFO]  server-10cb60c1-0099-a057-9905-5cbc25043093: entering follower state: follower="Node at 10cb60c1-0099-a057-9905-5cbc25043093 [Follower]" leader-address= leader-id=
2024-05-07T18:34:29.106Z [WARN]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:29.106Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: entering candidate state: node="Node at 3a86ff64-da9e-b194-198e-62c3ccff74da [Candidate]" term=2
2024-05-07T18:34:29.106Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: election won: term=2 tally=2
2024-05-07T18:34:29.106Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: entering leader state: leader="Node at 3a86ff64-da9e-b194-198e-62c3ccff74da [Leader]"
2024-05-07T18:34:29.106Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: added peer, starting replication: peer=server-ff8010cf-fe92-2260-1dea-910d58a048c8
2024-05-07T18:34:29.106Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: added peer, starting replication: peer=server-10cb60c1-0099-a057-9905-5cbc25043093
2024-05-07T18:34:29.106Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: pipelining replication: peer="{Voter server-10cb60c1-0099-a057-9905-5cbc25043093 10cb60c1-0099-a057-9905-5cbc25043093}"
2024-05-07T18:34:29.106Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: pipelining replication: peer="{Voter server-ff8010cf-fe92-2260-1dea-910d58a048c8 ff8010cf-fe92-2260-1dea-910d58a048c8}"
2024-05-07T18:34:29.214Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:29.106869167 +0000 UTC m=+6.707987407 (1 nodes), 58.799724ms from start of poll, 59.041584ms from cluster start. Timeout at 2024-05-07 18:34:29.214482752 +0000 UTC m=+6.815600992, 107.613585ms after stability
2024-05-07T18:34:29.215Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: aborting pipeline replication: peer="{Voter server-10cb60c1-0099-a057-9905-5cbc25043093 10cb60c1-0099-a057-9905-5cbc25043093}"
2024-05-07T18:34:29.215Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: aborting pipeline replication: peer="{Voter server-ff8010cf-fe92-2260-1dea-910d58a048c8 ff8010cf-fe92-2260-1dea-910d58a048c8}"
2024-05-07T18:34:29.216Z [INFO]  creating new snapshot: path=/tmp/raft785207878/snapshots/2-27-1715106869216.tmp
2024-05-07T18:34:29.222Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: starting restore from snapshot: id=2-27-1715106869216 last-index=27 last-term=2 size-in-bytes=168
2024-05-07T18:34:29.230Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: snapshot restore progress: id=2-27-1715106869216 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2024-05-07T18:34:29.230Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: restored from snapshot: id=2-27-1715106869216 last-index=27 last-term=2 size-in-bytes=168
2024-05-07T18:34:29.230Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a86ff64-da9e-b194-198e-62c3ccff74da Address:3a86ff64-da9e-b194-198e-62c3ccff74da} {Suffrage:Voter ID:server-ff8010cf-fe92-2260-1dea-910d58a048c8 Address:ff8010cf-fe92-2260-1dea-910d58a048c8} {Suffrage:Voter ID:server-10cb60c1-0099-a057-9905-5cbc25043093 Address:10cb60c1-0099-a057-9905-5cbc25043093}]"
2024-05-07T18:34:29.230Z [INFO]  creating new snapshot: path=/tmp/raft724550107/snapshots/2-27-1715106869230.tmp
2024-05-07T18:34:29.231Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: starting restore from snapshot: id=2-27-1715106869230 last-index=27 last-term=2 size-in-bytes=168
2024-05-07T18:34:29.231Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: snapshot restore progress: id=2-27-1715106869230 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2024-05-07T18:34:29.231Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: restored from snapshot: id=2-27-1715106869230 last-index=27 last-term=2 size-in-bytes=168
2024-05-07T18:34:29.231Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a86ff64-da9e-b194-198e-62c3ccff74da Address:3a86ff64-da9e-b194-198e-62c3ccff74da} {Suffrage:Voter ID:server-ff8010cf-fe92-2260-1dea-910d58a048c8 Address:ff8010cf-fe92-2260-1dea-910d58a048c8} {Suffrage:Voter ID:server-10cb60c1-0099-a057-9905-5cbc25043093 Address:10cb60c1-0099-a057-9905-5cbc25043093}]"
2024-05-07T18:34:29.231Z [INFO]  creating new snapshot: path=/tmp/raft2914589513/snapshots/2-27-1715106869231.tmp
2024-05-07T18:34:29.232Z [INFO]  server-10cb60c1-0099-a057-9905-5cbc25043093: starting restore from snapshot: id=2-27-1715106869231 last-index=27 last-term=2 size-in-bytes=168
2024-05-07T18:34:29.232Z [INFO]  server-10cb60c1-0099-a057-9905-5cbc25043093: snapshot restore progress: id=2-27-1715106869231 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2024-05-07T18:34:29.232Z [INFO]  server-10cb60c1-0099-a057-9905-5cbc25043093: restored from snapshot: id=2-27-1715106869231 last-index=27 last-term=2 size-in-bytes=168
2024-05-07T18:34:29.232Z [INFO]  server-10cb60c1-0099-a057-9905-5cbc25043093: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a86ff64-da9e-b194-198e-62c3ccff74da Address:3a86ff64-da9e-b194-198e-62c3ccff74da} {Suffrage:Voter ID:server-ff8010cf-fe92-2260-1dea-910d58a048c8 Address:ff8010cf-fe92-2260-1dea-910d58a048c8} {Suffrage:Voter ID:server-10cb60c1-0099-a057-9905-5cbc25043093 Address:10cb60c1-0099-a057-9905-5cbc25043093}]"
2024-05-07T18:34:29.232Z [INFO]  server-3a86ff64-da9e-b194-198e-62c3ccff74da: entering follower state: follower="Node at 3a86ff64-da9e-b194-198e-62c3ccff74da [Follower]" leader-address= leader-id=
2024-05-07T18:34:29.232Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: entering follower state: follower="Node at ff8010cf-fe92-2260-1dea-910d58a048c8 [Follower]" leader-address= leader-id=
2024-05-07T18:34:29.232Z [INFO]  server-10cb60c1-0099-a057-9905-5cbc25043093: entering follower state: follower="Node at 10cb60c1-0099-a057-9905-5cbc25043093 [Follower]" leader-address= leader-id=
2024-05-07T18:34:29.310Z [WARN]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:29.310Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: entering candidate state: node="Node at ff8010cf-fe92-2260-1dea-910d58a048c8 [Candidate]" term=3
2024-05-07T18:34:29.310Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: election won: term=3 tally=2
2024-05-07T18:34:29.310Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: entering leader state: leader="Node at ff8010cf-fe92-2260-1dea-910d58a048c8 [Leader]"
2024-05-07T18:34:29.310Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: added peer, starting replication: peer=server-3a86ff64-da9e-b194-198e-62c3ccff74da
2024-05-07T18:34:29.310Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: added peer, starting replication: peer=server-10cb60c1-0099-a057-9905-5cbc25043093
2024-05-07T18:34:29.310Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: pipelining replication: peer="{Voter server-10cb60c1-0099-a057-9905-5cbc25043093 10cb60c1-0099-a057-9905-5cbc25043093}"
2024-05-07T18:34:29.310Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: pipelining replication: peer="{Voter server-3a86ff64-da9e-b194-198e-62c3ccff74da 3a86ff64-da9e-b194-198e-62c3ccff74da}"
2024-05-07T18:34:29.554Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:29.674Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:29.554559356 +0000 UTC m=+7.155677596 (1 nodes), 28.062µs from start of poll, 506.731773ms from cluster start. Timeout at 2024-05-07 18:34:29.674493257 +0000 UTC m=+7.275611497, 119.933901ms after stability
2024-05-07T18:34:29.674Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: aborting pipeline replication: peer="{Voter server-10cb60c1-0099-a057-9905-5cbc25043093 10cb60c1-0099-a057-9905-5cbc25043093}"
2024-05-07T18:34:29.674Z [INFO]  server-ff8010cf-fe92-2260-1dea-910d58a048c8: aborting pipeline replication: peer="{Voter server-3a86ff64-da9e-b194-198e-62c3ccff74da 3a86ff64-da9e-b194-198e-62c3ccff74da}"
--- PASS: TestRaft_RecoverCluster (1.83s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.60s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.60s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.63s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2024-05-07T18:34:29.675Z [INFO]  server-66bb172d-ba5d-796e-1f92-12b76fd3adae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66bb172d-ba5d-796e-1f92-12b76fd3adae Address:66bb172d-ba5d-796e-1f92-12b76fd3adae}]"
2024-05-07T18:34:29.675Z [INFO]  server-66bb172d-ba5d-796e-1f92-12b76fd3adae: entering follower state: follower="Node at 66bb172d-ba5d-796e-1f92-12b76fd3adae [Follower]" leader-address= leader-id=
2024-05-07T18:34:29.762Z [WARN]  server-66bb172d-ba5d-796e-1f92-12b76fd3adae: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:29.762Z [INFO]  server-66bb172d-ba5d-796e-1f92-12b76fd3adae: entering candidate state: node="Node at 66bb172d-ba5d-796e-1f92-12b76fd3adae [Candidate]" term=2
2024-05-07T18:34:29.762Z [INFO]  server-66bb172d-ba5d-796e-1f92-12b76fd3adae: election won: term=2 tally=1
2024-05-07T18:34:29.762Z [INFO]  server-66bb172d-ba5d-796e-1f92-12b76fd3adae: entering leader state: leader="Node at 66bb172d-ba5d-796e-1f92-12b76fd3adae [Leader]"
--- PASS: TestRaft_SingleNode (0.09s)
=== RUN   TestRaft_TripleNode
2024-05-07T18:34:29.763Z [INFO]  server-759f8c62-7907-15f6-a6a2-c3d3740c7a0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-759f8c62-7907-15f6-a6a2-c3d3740c7a0d Address:759f8c62-7907-15f6-a6a2-c3d3740c7a0d} {Suffrage:Voter ID:server-8e35d200-7698-4cc5-7e69-dd450386107d Address:8e35d200-7698-4cc5-7e69-dd450386107d} {Suffrage:Voter ID:server-27513b13-f7bb-1243-f791-f92398bfbcc2 Address:27513b13-f7bb-1243-f791-f92398bfbcc2}]"
2024-05-07T18:34:29.763Z [INFO]  server-8e35d200-7698-4cc5-7e69-dd450386107d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-759f8c62-7907-15f6-a6a2-c3d3740c7a0d Address:759f8c62-7907-15f6-a6a2-c3d3740c7a0d} {Suffrage:Voter ID:server-8e35d200-7698-4cc5-7e69-dd450386107d Address:8e35d200-7698-4cc5-7e69-dd450386107d} {Suffrage:Voter ID:server-27513b13-f7bb-1243-f791-f92398bfbcc2 Address:27513b13-f7bb-1243-f791-f92398bfbcc2}]"
2024-05-07T18:34:29.763Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-759f8c62-7907-15f6-a6a2-c3d3740c7a0d Address:759f8c62-7907-15f6-a6a2-c3d3740c7a0d} {Suffrage:Voter ID:server-8e35d200-7698-4cc5-7e69-dd450386107d Address:8e35d200-7698-4cc5-7e69-dd450386107d} {Suffrage:Voter ID:server-27513b13-f7bb-1243-f791-f92398bfbcc2 Address:27513b13-f7bb-1243-f791-f92398bfbcc2}]"
2024-05-07T18:34:29.763Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:29.763Z [INFO]  server-759f8c62-7907-15f6-a6a2-c3d3740c7a0d: entering follower state: follower="Node at 759f8c62-7907-15f6-a6a2-c3d3740c7a0d [Follower]" leader-address= leader-id=
2024-05-07T18:34:29.763Z [INFO]  server-8e35d200-7698-4cc5-7e69-dd450386107d: entering follower state: follower="Node at 8e35d200-7698-4cc5-7e69-dd450386107d [Follower]" leader-address= leader-id=
2024-05-07T18:34:29.763Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: entering follower state: follower="Node at 27513b13-f7bb-1243-f791-f92398bfbcc2 [Follower]" leader-address= leader-id=
2024-05-07T18:34:29.830Z [WARN]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:29.830Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: entering candidate state: node="Node at 27513b13-f7bb-1243-f791-f92398bfbcc2 [Candidate]" term=2
2024-05-07T18:34:29.830Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: election won: term=2 tally=2
2024-05-07T18:34:29.830Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: entering leader state: leader="Node at 27513b13-f7bb-1243-f791-f92398bfbcc2 [Leader]"
2024-05-07T18:34:29.830Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: added peer, starting replication: peer=server-759f8c62-7907-15f6-a6a2-c3d3740c7a0d
2024-05-07T18:34:29.830Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: added peer, starting replication: peer=server-8e35d200-7698-4cc5-7e69-dd450386107d
2024-05-07T18:34:29.830Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: pipelining replication: peer="{Voter server-8e35d200-7698-4cc5-7e69-dd450386107d 8e35d200-7698-4cc5-7e69-dd450386107d}"
2024-05-07T18:34:29.830Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: pipelining replication: peer="{Voter server-759f8c62-7907-15f6-a6a2-c3d3740c7a0d 759f8c62-7907-15f6-a6a2-c3d3740c7a0d}"
2024-05-07T18:34:29.938Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:29.830841366 +0000 UTC m=+7.431959606 (2 nodes), 67.196912ms from start of poll, 67.420989ms from cluster start. Timeout at 2024-05-07 18:34:29.938499263 +0000 UTC m=+7.539617513, 107.657907ms after stability
2024-05-07T18:34:29.938Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:30.046Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:29.938603548 +0000 UTC m=+7.539721788 (1 nodes), 1.453µs from start of poll, 175.183171ms from cluster start. Timeout at 2024-05-07 18:34:30.046506341 +0000 UTC m=+7.647624591, 107.902803ms after stability
2024-05-07T18:34:30.062Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: aborting pipeline replication: peer="{Voter server-8e35d200-7698-4cc5-7e69-dd450386107d 8e35d200-7698-4cc5-7e69-dd450386107d}"
2024-05-07T18:34:30.062Z [INFO]  server-27513b13-f7bb-1243-f791-f92398bfbcc2: aborting pipeline replication: peer="{Voter server-759f8c62-7907-15f6-a6a2-c3d3740c7a0d 759f8c62-7907-15f6-a6a2-c3d3740c7a0d}"
--- PASS: TestRaft_TripleNode (0.30s)
=== RUN   TestRaft_LeaderFail
2024-05-07T18:34:30.063Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-721b25d1-076f-5c5c-7931-441b7bc80036 Address:721b25d1-076f-5c5c-7931-441b7bc80036} {Suffrage:Voter ID:server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e Address:d5c2d296-eef7-4856-e9bb-1d5a32e0046e} {Suffrage:Voter ID:server-20183e15-f6b6-1c9b-f62d-3d24605528c6 Address:20183e15-f6b6-1c9b-f62d-3d24605528c6}]"
2024-05-07T18:34:30.063Z [INFO]  server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-721b25d1-076f-5c5c-7931-441b7bc80036 Address:721b25d1-076f-5c5c-7931-441b7bc80036} {Suffrage:Voter ID:server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e Address:d5c2d296-eef7-4856-e9bb-1d5a32e0046e} {Suffrage:Voter ID:server-20183e15-f6b6-1c9b-f62d-3d24605528c6 Address:20183e15-f6b6-1c9b-f62d-3d24605528c6}]"
2024-05-07T18:34:30.063Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-721b25d1-076f-5c5c-7931-441b7bc80036 Address:721b25d1-076f-5c5c-7931-441b7bc80036} {Suffrage:Voter ID:server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e Address:d5c2d296-eef7-4856-e9bb-1d5a32e0046e} {Suffrage:Voter ID:server-20183e15-f6b6-1c9b-f62d-3d24605528c6 Address:20183e15-f6b6-1c9b-f62d-3d24605528c6}]"
2024-05-07T18:34:30.063Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:30.063Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: entering follower state: follower="Node at 721b25d1-076f-5c5c-7931-441b7bc80036 [Follower]" leader-address= leader-id=
2024-05-07T18:34:30.063Z [INFO]  server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e: entering follower state: follower="Node at d5c2d296-eef7-4856-e9bb-1d5a32e0046e [Follower]" leader-address= leader-id=
2024-05-07T18:34:30.063Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: entering follower state: follower="Node at 20183e15-f6b6-1c9b-f62d-3d24605528c6 [Follower]" leader-address= leader-id=
2024-05-07T18:34:30.154Z [WARN]  server-721b25d1-076f-5c5c-7931-441b7bc80036: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:30.154Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: entering candidate state: node="Node at 721b25d1-076f-5c5c-7931-441b7bc80036 [Candidate]" term=2
2024-05-07T18:34:30.154Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: election won: term=2 tally=2
2024-05-07T18:34:30.154Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: entering leader state: leader="Node at 721b25d1-076f-5c5c-7931-441b7bc80036 [Leader]"
2024-05-07T18:34:30.154Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: added peer, starting replication: peer=server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e
2024-05-07T18:34:30.154Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: added peer, starting replication: peer=server-20183e15-f6b6-1c9b-f62d-3d24605528c6
2024-05-07T18:34:30.154Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: pipelining replication: peer="{Voter server-20183e15-f6b6-1c9b-f62d-3d24605528c6 20183e15-f6b6-1c9b-f62d-3d24605528c6}"
2024-05-07T18:34:30.154Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: pipelining replication: peer="{Voter server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e d5c2d296-eef7-4856-e9bb-1d5a32e0046e}"
2024-05-07T18:34:30.262Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:30.154928981 +0000 UTC m=+7.756047221 (2 nodes), 91.19124ms from start of poll, 91.430766ms from cluster start. Timeout at 2024-05-07 18:34:30.262489057 +0000 UTC m=+7.863607297, 107.560076ms after stability
2024-05-07T18:34:30.262Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:30.370Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:30.262591868 +0000 UTC m=+7.863710108 (1 nodes), 1.493µs from start of poll, 199.093653ms from cluster start. Timeout at 2024-05-07 18:34:30.370481477 +0000 UTC m=+7.971599717, 107.889609ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at 721b25d1-076f-5c5c-7931-441b7bc80036 [Leader]
2024-05-07T18:34:30.386Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to pipeline appendEntries: peer="{Voter server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e d5c2d296-eef7-4856-e9bb-1d5a32e0046e}" error="append pipeline closed"
2024-05-07T18:34:30.386Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: aborting pipeline replication: peer="{Voter server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e d5c2d296-eef7-4856-e9bb-1d5a32e0046e}"
2024-05-07T18:34:30.390Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to heartbeat to: peer=d5c2d296-eef7-4856-e9bb-1d5a32e0046e backoff time=10ms error="failed to connect to peer: d5c2d296-eef7-4856-e9bb-1d5a32e0046e"
2024-05-07T18:34:30.390Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:30.390Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to heartbeat to: peer=20183e15-f6b6-1c9b-f62d-3d24605528c6 backoff time=10ms error="failed to connect to peer: 20183e15-f6b6-1c9b-f62d-3d24605528c6"
2024-05-07T18:34:30.394Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to appendEntries to: peer="{Voter server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e d5c2d296-eef7-4856-e9bb-1d5a32e0046e}" error="failed to connect to peer: d5c2d296-eef7-4856-e9bb-1d5a32e0046e"
2024-05-07T18:34:30.394Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to pipeline appendEntries: peer="{Voter server-20183e15-f6b6-1c9b-f62d-3d24605528c6 20183e15-f6b6-1c9b-f62d-3d24605528c6}" error="append pipeline closed"
2024-05-07T18:34:30.394Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: aborting pipeline replication: peer="{Voter server-20183e15-f6b6-1c9b-f62d-3d24605528c6 20183e15-f6b6-1c9b-f62d-3d24605528c6}"
2024-05-07T18:34:30.402Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to appendEntries to: peer="{Voter server-20183e15-f6b6-1c9b-f62d-3d24605528c6 20183e15-f6b6-1c9b-f62d-3d24605528c6}" error="failed to connect to peer: 20183e15-f6b6-1c9b-f62d-3d24605528c6"
2024-05-07T18:34:30.414Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to appendEntries to: peer="{Voter server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e d5c2d296-eef7-4856-e9bb-1d5a32e0046e}" error="failed to connect to peer: d5c2d296-eef7-4856-e9bb-1d5a32e0046e"
2024-05-07T18:34:30.418Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to heartbeat to: peer=d5c2d296-eef7-4856-e9bb-1d5a32e0046e backoff time=10ms error="failed to connect to peer: d5c2d296-eef7-4856-e9bb-1d5a32e0046e"
2024-05-07T18:34:30.418Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to heartbeat to: peer=20183e15-f6b6-1c9b-f62d-3d24605528c6 backoff time=10ms error="failed to connect to peer: 20183e15-f6b6-1c9b-f62d-3d24605528c6"
2024-05-07T18:34:30.426Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to appendEntries to: peer="{Voter server-20183e15-f6b6-1c9b-f62d-3d24605528c6 20183e15-f6b6-1c9b-f62d-3d24605528c6}" error="failed to connect to peer: 20183e15-f6b6-1c9b-f62d-3d24605528c6"
2024-05-07T18:34:30.434Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to appendEntries to: peer="{Voter server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e d5c2d296-eef7-4856-e9bb-1d5a32e0046e}" error="failed to connect to peer: d5c2d296-eef7-4856-e9bb-1d5a32e0046e"
2024-05-07T18:34:30.434Z [WARN]  server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to contact: server-id=server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e time=52.034211ms
2024-05-07T18:34:30.434Z [WARN]  server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to contact: server-id=server-20183e15-f6b6-1c9b-f62d-3d24605528c6 time=51.977365ms
2024-05-07T18:34:30.434Z [WARN]  server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to contact quorum of nodes, stepping down
2024-05-07T18:34:30.434Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: entering follower state: follower="Node at 721b25d1-076f-5c5c-7931-441b7bc80036 [Follower]" leader-address= leader-id=
2024-05-07T18:34:30.446Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to appendEntries to: peer="{Voter server-20183e15-f6b6-1c9b-f62d-3d24605528c6 20183e15-f6b6-1c9b-f62d-3d24605528c6}" error="failed to connect to peer: 20183e15-f6b6-1c9b-f62d-3d24605528c6"
2024-05-07T18:34:30.446Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to heartbeat to: peer=20183e15-f6b6-1c9b-f62d-3d24605528c6 backoff time=10ms error="failed to connect to peer: 20183e15-f6b6-1c9b-f62d-3d24605528c6"
2024-05-07T18:34:30.478Z [WARN]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: heartbeat timeout reached, starting election: last-leader-addr=721b25d1-076f-5c5c-7931-441b7bc80036 last-leader-id=server-721b25d1-076f-5c5c-7931-441b7bc80036
2024-05-07T18:34:30.478Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: entering candidate state: node="Node at 20183e15-f6b6-1c9b-f62d-3d24605528c6 [Candidate]" term=3
2024-05-07T18:34:30.478Z [WARN]  server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e: rejecting vote request since we have a leader: from=20183e15-f6b6-1c9b-f62d-3d24605528c6 leader=721b25d1-076f-5c5c-7931-441b7bc80036 leader-id=server-721b25d1-076f-5c5c-7931-441b7bc80036
2024-05-07T18:34:30.478Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to make requestVote RPC: target="{Voter server-721b25d1-076f-5c5c-7931-441b7bc80036 721b25d1-076f-5c5c-7931-441b7bc80036}" error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036" term=3
2024-05-07T18:34:30.506Z [WARN]  server-721b25d1-076f-5c5c-7931-441b7bc80036: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:30.506Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: entering candidate state: node="Node at 721b25d1-076f-5c5c-7931-441b7bc80036 [Candidate]" term=3
2024-05-07T18:34:30.506Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to make requestVote RPC: target="{Voter server-20183e15-f6b6-1c9b-f62d-3d24605528c6 20183e15-f6b6-1c9b-f62d-3d24605528c6}" error="failed to connect to peer: 20183e15-f6b6-1c9b-f62d-3d24605528c6" term=3
2024-05-07T18:34:30.506Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to make requestVote RPC: target="{Voter server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e d5c2d296-eef7-4856-e9bb-1d5a32e0046e}" error="failed to connect to peer: d5c2d296-eef7-4856-e9bb-1d5a32e0046e" term=3
2024-05-07T18:34:30.518Z [WARN]  server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e: heartbeat timeout reached, starting election: last-leader-addr=721b25d1-076f-5c5c-7931-441b7bc80036 last-leader-id=server-721b25d1-076f-5c5c-7931-441b7bc80036
2024-05-07T18:34:30.518Z [INFO]  server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e: entering candidate state: node="Node at d5c2d296-eef7-4856-e9bb-1d5a32e0046e [Candidate]" term=3
2024-05-07T18:34:30.518Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: duplicate requestVote for same term: term=3
2024-05-07T18:34:30.518Z [ERROR] server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e: failed to make requestVote RPC: target="{Voter server-721b25d1-076f-5c5c-7931-441b7bc80036 721b25d1-076f-5c5c-7931-441b7bc80036}" error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036" term=3
2024-05-07T18:34:30.550Z [WARN]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: Election timeout reached, restarting election
2024-05-07T18:34:30.550Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: entering candidate state: node="Node at 20183e15-f6b6-1c9b-f62d-3d24605528c6 [Candidate]" term=4
2024-05-07T18:34:30.550Z [INFO]  server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e: entering follower state: follower="Node at d5c2d296-eef7-4856-e9bb-1d5a32e0046e [Follower]" leader-address= leader-id=
2024-05-07T18:34:30.550Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: election won: term=4 tally=2
2024-05-07T18:34:30.550Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: entering leader state: leader="Node at 20183e15-f6b6-1c9b-f62d-3d24605528c6 [Leader]"
2024-05-07T18:34:30.550Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: added peer, starting replication: peer=server-721b25d1-076f-5c5c-7931-441b7bc80036
2024-05-07T18:34:30.550Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: added peer, starting replication: peer=server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e
2024-05-07T18:34:30.550Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: pipelining replication: peer="{Voter server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e d5c2d296-eef7-4856-e9bb-1d5a32e0046e}"
2024-05-07T18:34:30.550Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to make requestVote RPC: target="{Voter server-721b25d1-076f-5c5c-7931-441b7bc80036 721b25d1-076f-5c5c-7931-441b7bc80036}" error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036" term=4
2024-05-07T18:34:30.550Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to appendEntries to: peer="{Voter server-721b25d1-076f-5c5c-7931-441b7bc80036 721b25d1-076f-5c5c-7931-441b7bc80036}" error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036"
2024-05-07T18:34:30.558Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to heartbeat to: peer=721b25d1-076f-5c5c-7931-441b7bc80036 backoff time=10ms error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036"
2024-05-07T18:34:30.570Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to appendEntries to: peer="{Voter server-721b25d1-076f-5c5c-7931-441b7bc80036 721b25d1-076f-5c5c-7931-441b7bc80036}" error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036"
2024-05-07T18:34:30.578Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to heartbeat to: peer=721b25d1-076f-5c5c-7931-441b7bc80036 backoff time=10ms error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036"
2024-05-07T18:34:30.590Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to appendEntries to: peer="{Voter server-721b25d1-076f-5c5c-7931-441b7bc80036 721b25d1-076f-5c5c-7931-441b7bc80036}" error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036"
2024-05-07T18:34:30.594Z [WARN]  server-721b25d1-076f-5c5c-7931-441b7bc80036: Election timeout reached, restarting election
2024-05-07T18:34:30.594Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: entering candidate state: node="Node at 721b25d1-076f-5c5c-7931-441b7bc80036 [Candidate]" term=4
2024-05-07T18:34:30.594Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to make requestVote RPC: target="{Voter server-20183e15-f6b6-1c9b-f62d-3d24605528c6 20183e15-f6b6-1c9b-f62d-3d24605528c6}" error="failed to connect to peer: 20183e15-f6b6-1c9b-f62d-3d24605528c6" term=4
2024-05-07T18:34:30.594Z [ERROR] server-721b25d1-076f-5c5c-7931-441b7bc80036: failed to make requestVote RPC: target="{Voter server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e d5c2d296-eef7-4856-e9bb-1d5a32e0046e}" error="failed to connect to peer: d5c2d296-eef7-4856-e9bb-1d5a32e0046e" term=4
2024-05-07T18:34:30.598Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to heartbeat to: peer=721b25d1-076f-5c5c-7931-441b7bc80036 backoff time=10ms error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036"
2024-05-07T18:34:30.602Z [WARN]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to contact: server-id=server-721b25d1-076f-5c5c-7931-441b7bc80036 time=51.873472ms
2024-05-07T18:34:30.622Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to appendEntries to: peer="{Voter server-721b25d1-076f-5c5c-7931-441b7bc80036 721b25d1-076f-5c5c-7931-441b7bc80036}" error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036"
2024-05-07T18:34:30.626Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to heartbeat to: peer=721b25d1-076f-5c5c-7931-441b7bc80036 backoff time=20ms error="failed to connect to peer: 721b25d1-076f-5c5c-7931-441b7bc80036"
2024-05-07T18:34:30.650Z [WARN]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to contact: server-id=server-721b25d1-076f-5c5c-7931-441b7bc80036 time=99.920035ms
2024-05-07T18:34:30.656Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:30.550718158 +0000 UTC m=+8.151836398 (1 nodes), 160.207838ms from start of poll, 487.219943ms from cluster start. Timeout at 2024-05-07 18:34:30.656731707 +0000 UTC m=+8.257849947, 106.013549ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at 721b25d1-076f-5c5c-7931-441b7bc80036 [Candidate]
2024-05-07T18:34:30.657Z [INFO]  server-721b25d1-076f-5c5c-7931-441b7bc80036: entering follower state: follower="Node at 721b25d1-076f-5c5c-7931-441b7bc80036 [Follower]" leader-address=20183e15-f6b6-1c9b-f62d-3d24605528c6 leader-id=server-20183e15-f6b6-1c9b-f62d-3d24605528c6
2024-05-07T18:34:30.670Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: pipelining replication: peer="{Voter server-721b25d1-076f-5c5c-7931-441b7bc80036 721b25d1-076f-5c5c-7931-441b7bc80036}"
2024-05-07T18:34:30.670Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: aborting pipeline replication: peer="{Voter server-d5c2d296-eef7-4856-e9bb-1d5a32e0046e d5c2d296-eef7-4856-e9bb-1d5a32e0046e}"
2024-05-07T18:34:30.670Z [INFO]  server-20183e15-f6b6-1c9b-f62d-3d24605528c6: aborting pipeline replication: peer="{Voter server-721b25d1-076f-5c5c-7931-441b7bc80036 721b25d1-076f-5c5c-7931-441b7bc80036}"
2024-05-07T18:34:31.171Z [ERROR] server-20183e15-f6b6-1c9b-f62d-3d24605528c6: failed to heartbeat to: peer=721b25d1-076f-5c5c-7931-441b7bc80036 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeaderFail (1.11s)
=== RUN   TestRaft_BehindFollower
2024-05-07T18:34:31.172Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31663476-9aaf-fc87-4b84-04836909411c Address:31663476-9aaf-fc87-4b84-04836909411c} {Suffrage:Voter ID:server-e6bd671e-0432-7844-7863-3a966f2ba625 Address:e6bd671e-0432-7844-7863-3a966f2ba625} {Suffrage:Voter ID:server-371f851c-7134-ed02-96a9-7e8a9ba74409 Address:371f851c-7134-ed02-96a9-7e8a9ba74409}]"
2024-05-07T18:34:31.172Z [INFO]  server-e6bd671e-0432-7844-7863-3a966f2ba625: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31663476-9aaf-fc87-4b84-04836909411c Address:31663476-9aaf-fc87-4b84-04836909411c} {Suffrage:Voter ID:server-e6bd671e-0432-7844-7863-3a966f2ba625 Address:e6bd671e-0432-7844-7863-3a966f2ba625} {Suffrage:Voter ID:server-371f851c-7134-ed02-96a9-7e8a9ba74409 Address:371f851c-7134-ed02-96a9-7e8a9ba74409}]"
2024-05-07T18:34:31.172Z [INFO]  server-371f851c-7134-ed02-96a9-7e8a9ba74409: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31663476-9aaf-fc87-4b84-04836909411c Address:31663476-9aaf-fc87-4b84-04836909411c} {Suffrage:Voter ID:server-e6bd671e-0432-7844-7863-3a966f2ba625 Address:e6bd671e-0432-7844-7863-3a966f2ba625} {Suffrage:Voter ID:server-371f851c-7134-ed02-96a9-7e8a9ba74409 Address:371f851c-7134-ed02-96a9-7e8a9ba74409}]"
2024-05-07T18:34:31.172Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:31.172Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: entering follower state: follower="Node at 31663476-9aaf-fc87-4b84-04836909411c [Follower]" leader-address= leader-id=
2024-05-07T18:34:31.172Z [INFO]  server-e6bd671e-0432-7844-7863-3a966f2ba625: entering follower state: follower="Node at e6bd671e-0432-7844-7863-3a966f2ba625 [Follower]" leader-address= leader-id=
2024-05-07T18:34:31.172Z [INFO]  server-371f851c-7134-ed02-96a9-7e8a9ba74409: entering follower state: follower="Node at 371f851c-7134-ed02-96a9-7e8a9ba74409 [Follower]" leader-address= leader-id=
2024-05-07T18:34:31.229Z [WARN]  server-31663476-9aaf-fc87-4b84-04836909411c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:31.229Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: entering candidate state: node="Node at 31663476-9aaf-fc87-4b84-04836909411c [Candidate]" term=2
2024-05-07T18:34:31.229Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: election won: term=2 tally=2
2024-05-07T18:34:31.229Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: entering leader state: leader="Node at 31663476-9aaf-fc87-4b84-04836909411c [Leader]"
2024-05-07T18:34:31.229Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: added peer, starting replication: peer=server-e6bd671e-0432-7844-7863-3a966f2ba625
2024-05-07T18:34:31.229Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: added peer, starting replication: peer=server-371f851c-7134-ed02-96a9-7e8a9ba74409
2024-05-07T18:34:31.229Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: pipelining replication: peer="{Voter server-371f851c-7134-ed02-96a9-7e8a9ba74409 371f851c-7134-ed02-96a9-7e8a9ba74409}"
2024-05-07T18:34:31.229Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: pipelining replication: peer="{Voter server-e6bd671e-0432-7844-7863-3a966f2ba625 e6bd671e-0432-7844-7863-3a966f2ba625}"
2024-05-07T18:34:31.338Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:31.229634837 +0000 UTC m=+8.830753077 (1 nodes), 56.744272ms from start of poll, 56.961646ms from cluster start. Timeout at 2024-05-07 18:34:31.338505129 +0000 UTC m=+8.939623389, 108.870312ms after stability
2024-05-07T18:34:31.338Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:31.446Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:31.338596879 +0000 UTC m=+8.939715139 (2 nodes), 2.594µs from start of poll, 165.923708ms from cluster start. Timeout at 2024-05-07 18:34:31.446478233 +0000 UTC m=+9.047596473, 107.881334ms after stability
2024-05-07T18:34:31.446Z [ERROR] server-31663476-9aaf-fc87-4b84-04836909411c: failed to pipeline appendEntries: peer="{Voter server-e6bd671e-0432-7844-7863-3a966f2ba625 e6bd671e-0432-7844-7863-3a966f2ba625}" error="append pipeline closed"
2024-05-07T18:34:31.446Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: aborting pipeline replication: peer="{Voter server-e6bd671e-0432-7844-7863-3a966f2ba625 e6bd671e-0432-7844-7863-3a966f2ba625}"
2024-05-07T18:34:31.446Z [ERROR] server-31663476-9aaf-fc87-4b84-04836909411c: failed to appendEntries to: peer="{Voter server-e6bd671e-0432-7844-7863-3a966f2ba625 e6bd671e-0432-7844-7863-3a966f2ba625}" error="failed to connect to peer: e6bd671e-0432-7844-7863-3a966f2ba625"
2024-05-07T18:34:31.458Z [ERROR] server-31663476-9aaf-fc87-4b84-04836909411c: failed to heartbeat to: peer=e6bd671e-0432-7844-7863-3a966f2ba625 backoff time=10ms error="failed to connect to peer: e6bd671e-0432-7844-7863-3a966f2ba625"
2024-05-07T18:34:31.459Z [ERROR] server-31663476-9aaf-fc87-4b84-04836909411c: failed to appendEntries to: peer="{Voter server-e6bd671e-0432-7844-7863-3a966f2ba625 e6bd671e-0432-7844-7863-3a966f2ba625}" error="failed to connect to peer: e6bd671e-0432-7844-7863-3a966f2ba625"
    raft_test.go:459: [INFO] Finished apply without behind follower
2024-05-07T18:34:31.470Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: pipelining replication: peer="{Voter server-e6bd671e-0432-7844-7863-3a966f2ba625 e6bd671e-0432-7844-7863-3a966f2ba625}"
2024-05-07T18:34:31.478Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:31.586Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:31.478570192 +0000 UTC m=+9.079688432 (1 nodes), 1.713µs from start of poll, 305.897001ms from cluster start. Timeout at 2024-05-07 18:34:31.586487022 +0000 UTC m=+9.187605272, 107.91684ms after stability
2024-05-07T18:34:31.586Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: aborting pipeline replication: peer="{Voter server-371f851c-7134-ed02-96a9-7e8a9ba74409 371f851c-7134-ed02-96a9-7e8a9ba74409}"
2024-05-07T18:34:31.586Z [INFO]  server-31663476-9aaf-fc87-4b84-04836909411c: aborting pipeline replication: peer="{Voter server-e6bd671e-0432-7844-7863-3a966f2ba625 e6bd671e-0432-7844-7863-3a966f2ba625}"
--- PASS: TestRaft_BehindFollower (0.41s)
=== RUN   TestRaft_ApplyNonLeader
2024-05-07T18:34:31.588Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f Address:6b5a0759-faab-d5cf-3fb7-4b1f79680e7f} {Suffrage:Voter ID:server-c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84 Address:c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84} {Suffrage:Voter ID:server-ddcd6894-50e4-c427-92c5-411a0f3fe05c Address:ddcd6894-50e4-c427-92c5-411a0f3fe05c}]"
2024-05-07T18:34:31.588Z [INFO]  server-c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f Address:6b5a0759-faab-d5cf-3fb7-4b1f79680e7f} {Suffrage:Voter ID:server-c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84 Address:c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84} {Suffrage:Voter ID:server-ddcd6894-50e4-c427-92c5-411a0f3fe05c Address:ddcd6894-50e4-c427-92c5-411a0f3fe05c}]"
2024-05-07T18:34:31.588Z [INFO]  server-ddcd6894-50e4-c427-92c5-411a0f3fe05c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f Address:6b5a0759-faab-d5cf-3fb7-4b1f79680e7f} {Suffrage:Voter ID:server-c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84 Address:c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84} {Suffrage:Voter ID:server-ddcd6894-50e4-c427-92c5-411a0f3fe05c Address:ddcd6894-50e4-c427-92c5-411a0f3fe05c}]"
2024-05-07T18:34:31.588Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:31.588Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: entering follower state: follower="Node at 6b5a0759-faab-d5cf-3fb7-4b1f79680e7f [Follower]" leader-address= leader-id=
2024-05-07T18:34:31.588Z [INFO]  server-c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84: entering follower state: follower="Node at c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84 [Follower]" leader-address= leader-id=
2024-05-07T18:34:31.588Z [INFO]  server-ddcd6894-50e4-c427-92c5-411a0f3fe05c: entering follower state: follower="Node at ddcd6894-50e4-c427-92c5-411a0f3fe05c [Follower]" leader-address= leader-id=
2024-05-07T18:34:31.650Z [WARN]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:31.650Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: entering candidate state: node="Node at 6b5a0759-faab-d5cf-3fb7-4b1f79680e7f [Candidate]" term=2
2024-05-07T18:34:31.650Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: election won: term=2 tally=2
2024-05-07T18:34:31.650Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: entering leader state: leader="Node at 6b5a0759-faab-d5cf-3fb7-4b1f79680e7f [Leader]"
2024-05-07T18:34:31.650Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: added peer, starting replication: peer=server-c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84
2024-05-07T18:34:31.650Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: added peer, starting replication: peer=server-ddcd6894-50e4-c427-92c5-411a0f3fe05c
2024-05-07T18:34:31.650Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: pipelining replication: peer="{Voter server-ddcd6894-50e4-c427-92c5-411a0f3fe05c ddcd6894-50e4-c427-92c5-411a0f3fe05c}"
2024-05-07T18:34:31.651Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: pipelining replication: peer="{Voter server-c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84 c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84}"
2024-05-07T18:34:31.758Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:31.651129504 +0000 UTC m=+9.252247764 (1 nodes), 62.924647ms from start of poll, 63.206391ms from cluster start. Timeout at 2024-05-07 18:34:31.758500578 +0000 UTC m=+9.359618838, 107.371074ms after stability
2024-05-07T18:34:31.758Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:31.866Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:31.758590395 +0000 UTC m=+9.359708655 (2 nodes), 1.823µs from start of poll, 170.667282ms from cluster start. Timeout at 2024-05-07 18:34:31.866480214 +0000 UTC m=+9.467598474, 107.889819ms after stability
2024-05-07T18:34:31.866Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: aborting pipeline replication: peer="{Voter server-ddcd6894-50e4-c427-92c5-411a0f3fe05c ddcd6894-50e4-c427-92c5-411a0f3fe05c}"
2024-05-07T18:34:31.866Z [INFO]  server-6b5a0759-faab-d5cf-3fb7-4b1f79680e7f: aborting pipeline replication: peer="{Voter server-c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84 c06bf8eb-e9ff-d6ea-d9c9-3b884426ee84}"
--- PASS: TestRaft_ApplyNonLeader (0.28s)
=== RUN   TestRaft_ApplyConcurrent
2024-05-07T18:34:31.867Z [INFO]  server-991f38bf-60ad-a602-9a49-468db6ee9b23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-991f38bf-60ad-a602-9a49-468db6ee9b23 Address:991f38bf-60ad-a602-9a49-468db6ee9b23} {Suffrage:Voter ID:server-1ad1e0a8-071b-ea75-1057-da784034c3c1 Address:1ad1e0a8-071b-ea75-1057-da784034c3c1} {Suffrage:Voter ID:server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc Address:e2713fe3-4b8c-d5ea-ba61-11006722e6bc}]"
2024-05-07T18:34:31.867Z [INFO]  server-1ad1e0a8-071b-ea75-1057-da784034c3c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-991f38bf-60ad-a602-9a49-468db6ee9b23 Address:991f38bf-60ad-a602-9a49-468db6ee9b23} {Suffrage:Voter ID:server-1ad1e0a8-071b-ea75-1057-da784034c3c1 Address:1ad1e0a8-071b-ea75-1057-da784034c3c1} {Suffrage:Voter ID:server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc Address:e2713fe3-4b8c-d5ea-ba61-11006722e6bc}]"
2024-05-07T18:34:31.867Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-991f38bf-60ad-a602-9a49-468db6ee9b23 Address:991f38bf-60ad-a602-9a49-468db6ee9b23} {Suffrage:Voter ID:server-1ad1e0a8-071b-ea75-1057-da784034c3c1 Address:1ad1e0a8-071b-ea75-1057-da784034c3c1} {Suffrage:Voter ID:server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc Address:e2713fe3-4b8c-d5ea-ba61-11006722e6bc}]"
2024-05-07T18:34:31.867Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:31.868Z [INFO]  server-991f38bf-60ad-a602-9a49-468db6ee9b23: entering follower state: follower="Node at 991f38bf-60ad-a602-9a49-468db6ee9b23 [Follower]" leader-address= leader-id=
2024-05-07T18:34:31.868Z [INFO]  server-1ad1e0a8-071b-ea75-1057-da784034c3c1: entering follower state: follower="Node at 1ad1e0a8-071b-ea75-1057-da784034c3c1 [Follower]" leader-address= leader-id=
2024-05-07T18:34:31.868Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: entering follower state: follower="Node at e2713fe3-4b8c-d5ea-ba61-11006722e6bc [Follower]" leader-address= leader-id=
2024-05-07T18:34:32.050Z [WARN]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:32.050Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: entering candidate state: node="Node at e2713fe3-4b8c-d5ea-ba61-11006722e6bc [Candidate]" term=2
2024-05-07T18:34:32.050Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: election won: term=2 tally=2
2024-05-07T18:34:32.050Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: entering leader state: leader="Node at e2713fe3-4b8c-d5ea-ba61-11006722e6bc [Leader]"
2024-05-07T18:34:32.050Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: added peer, starting replication: peer=server-991f38bf-60ad-a602-9a49-468db6ee9b23
2024-05-07T18:34:32.050Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: added peer, starting replication: peer=server-1ad1e0a8-071b-ea75-1057-da784034c3c1
2024-05-07T18:34:32.050Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: pipelining replication: peer="{Voter server-1ad1e0a8-071b-ea75-1057-da784034c3c1 1ad1e0a8-071b-ea75-1057-da784034c3c1}"
2024-05-07T18:34:32.051Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: pipelining replication: peer="{Voter server-991f38bf-60ad-a602-9a49-468db6ee9b23 991f38bf-60ad-a602-9a49-468db6ee9b23}"
2024-05-07T18:34:32.258Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:32.051067487 +0000 UTC m=+9.652185747 (1 nodes), 183.082444ms from start of poll, 183.350993ms from cluster start. Timeout at 2024-05-07 18:34:32.258492556 +0000 UTC m=+9.859610797, 207.42505ms after stability
2024-05-07T18:34:32.310Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: aborting pipeline replication: peer="{Voter server-991f38bf-60ad-a602-9a49-468db6ee9b23 991f38bf-60ad-a602-9a49-468db6ee9b23}"
2024-05-07T18:34:32.310Z [INFO]  server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: aborting pipeline replication: peer="{Voter server-1ad1e0a8-071b-ea75-1057-da784034c3c1 1ad1e0a8-071b-ea75-1057-da784034c3c1}"
2024-05-07T18:34:32.814Z [ERROR] server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: failed to heartbeat to: peer=1ad1e0a8-071b-ea75-1057-da784034c3c1 backoff time=10ms error="command timed out"
2024-05-07T18:34:32.814Z [ERROR] server-e2713fe3-4b8c-d5ea-ba61-11006722e6bc: failed to heartbeat to: peer=991f38bf-60ad-a602-9a49-468db6ee9b23 backoff time=10ms error="command timed out"
--- PASS: TestRaft_ApplyConcurrent (0.95s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2024-05-07T18:34:32.815Z [INFO]  server-9ba0215a-68f8-cb36-8eb4-763ca20d833e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ba0215a-68f8-cb36-8eb4-763ca20d833e Address:9ba0215a-68f8-cb36-8eb4-763ca20d833e}]"
2024-05-07T18:34:32.815Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:32.815Z [INFO]  server-9ba0215a-68f8-cb36-8eb4-763ca20d833e: entering follower state: follower="Node at 9ba0215a-68f8-cb36-8eb4-763ca20d833e [Follower]" leader-address= leader-id=
2024-05-07T18:34:32.990Z [WARN]  server-9ba0215a-68f8-cb36-8eb4-763ca20d833e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:32.990Z [INFO]  server-9ba0215a-68f8-cb36-8eb4-763ca20d833e: entering candidate state: node="Node at 9ba0215a-68f8-cb36-8eb4-763ca20d833e [Candidate]" term=2
2024-05-07T18:34:32.990Z [INFO]  server-9ba0215a-68f8-cb36-8eb4-763ca20d833e: election won: term=2 tally=1
2024-05-07T18:34:32.990Z [INFO]  server-9ba0215a-68f8-cb36-8eb4-763ca20d833e: entering leader state: leader="Node at 9ba0215a-68f8-cb36-8eb4-763ca20d833e [Leader]"
2024-05-07T18:34:33.194Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:32.990690685 +0000 UTC m=+10.591808925 (1 nodes), 175.53177ms from start of poll, 175.630343ms from cluster start. Timeout at 2024-05-07 18:34:33.194502093 +0000 UTC m=+10.795620334, 203.811409ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.38s)
=== RUN   TestRaft_JoinNode
2024-05-07T18:34:33.196Z [INFO]  server-c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3 Address:c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3} {Suffrage:Voter ID:server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38 Address:7d7b413e-6baa-adf9-9bc7-c8dda8132a38}]"
2024-05-07T18:34:33.196Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3 Address:c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3} {Suffrage:Voter ID:server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38 Address:7d7b413e-6baa-adf9-9bc7-c8dda8132a38}]"
2024-05-07T18:34:33.196Z [INFO]  server-8ec9c256-3d15-a4e1-7a19-25cdcabf4db0: initial configuration: index=0 servers=[]
2024-05-07T18:34:33.196Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:33.196Z [INFO]  server-c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3: entering follower state: follower="Node at c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3 [Follower]" leader-address= leader-id=
2024-05-07T18:34:33.196Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: entering follower state: follower="Node at 7d7b413e-6baa-adf9-9bc7-c8dda8132a38 [Follower]" leader-address= leader-id=
2024-05-07T18:34:33.196Z [INFO]  server-8ec9c256-3d15-a4e1-7a19-25cdcabf4db0: entering follower state: follower="Node at 8ec9c256-3d15-a4e1-7a19-25cdcabf4db0 [Follower]" leader-address= leader-id=
2024-05-07T18:34:33.250Z [WARN]  server-8ec9c256-3d15-a4e1-7a19-25cdcabf4db0: no known peers, aborting election
2024-05-07T18:34:33.294Z [WARN]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:33.294Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: entering candidate state: node="Node at 7d7b413e-6baa-adf9-9bc7-c8dda8132a38 [Candidate]" term=2
2024-05-07T18:34:33.294Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: election won: term=2 tally=2
2024-05-07T18:34:33.294Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: entering leader state: leader="Node at 7d7b413e-6baa-adf9-9bc7-c8dda8132a38 [Leader]"
2024-05-07T18:34:33.294Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: added peer, starting replication: peer=server-c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3
2024-05-07T18:34:33.294Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: pipelining replication: peer="{Voter server-c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3 c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3}"
2024-05-07T18:34:33.402Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:33.294745521 +0000 UTC m=+10.895863761 (1 nodes), 98.230925ms from start of poll, 98.547373ms from cluster start. Timeout at 2024-05-07 18:34:33.402544401 +0000 UTC m=+11.003662651, 107.79889ms after stability
2024-05-07T18:34:33.402Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: updating configuration: command=AddVoter server-id=server-8ec9c256-3d15-a4e1-7a19-25cdcabf4db0 server-addr=8ec9c256-3d15-a4e1-7a19-25cdcabf4db0 servers="[{Suffrage:Voter ID:server-c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3 Address:c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3} {Suffrage:Voter ID:server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38 Address:7d7b413e-6baa-adf9-9bc7-c8dda8132a38} {Suffrage:Voter ID:server-8ec9c256-3d15-a4e1-7a19-25cdcabf4db0 Address:8ec9c256-3d15-a4e1-7a19-25cdcabf4db0}]"
2024-05-07T18:34:33.402Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: added peer, starting replication: peer=server-8ec9c256-3d15-a4e1-7a19-25cdcabf4db0
2024-05-07T18:34:33.402Z [WARN]  server-8ec9c256-3d15-a4e1-7a19-25cdcabf4db0: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2024-05-07T18:34:33.402Z [WARN]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: appendEntries rejected, sending older logs: peer="{Voter server-8ec9c256-3d15-a4e1-7a19-25cdcabf4db0 8ec9c256-3d15-a4e1-7a19-25cdcabf4db0}" next=1
2024-05-07T18:34:33.402Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: pipelining replication: peer="{Voter server-8ec9c256-3d15-a4e1-7a19-25cdcabf4db0 8ec9c256-3d15-a4e1-7a19-25cdcabf4db0}"
2024-05-07T18:34:33.402Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:33.510Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:33.402832918 +0000 UTC m=+11.003951158 (1 nodes), 591ns from start of poll, 206.63477ms from cluster start. Timeout at 2024-05-07 18:34:33.510485756 +0000 UTC m=+11.111603996, 107.652838ms after stability
2024-05-07T18:34:33.510Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: aborting pipeline replication: peer="{Voter server-8ec9c256-3d15-a4e1-7a19-25cdcabf4db0 8ec9c256-3d15-a4e1-7a19-25cdcabf4db0}"
2024-05-07T18:34:33.510Z [INFO]  server-7d7b413e-6baa-adf9-9bc7-c8dda8132a38: aborting pipeline replication: peer="{Voter server-c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3 c2910e01-7ca8-b72a-e2ea-d28d0cf91fe3}"
--- PASS: TestRaft_JoinNode (0.31s)
=== RUN   TestRaft_JoinNode_ConfigStore
2024-05-07T18:34:33.511Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b258852e-dc02-0129-f1a7-850343ca1f4c Address:b258852e-dc02-0129-f1a7-850343ca1f4c}]"
2024-05-07T18:34:33.511Z [INFO]  server-c810e15b-952b-59bb-350b-b41e8fdb918d: initial configuration: index=0 servers=[]
2024-05-07T18:34:33.511Z [INFO]  server-38746261-7d44-fcc9-45f3-8d91eb0a1f74: initial configuration: index=0 servers=[]
2024-05-07T18:34:33.511Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:33.511Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: entering follower state: follower="Node at b258852e-dc02-0129-f1a7-850343ca1f4c [Follower]" leader-address= leader-id=
2024-05-07T18:34:33.511Z [INFO]  server-c810e15b-952b-59bb-350b-b41e8fdb918d: entering follower state: follower="Node at c810e15b-952b-59bb-350b-b41e8fdb918d [Follower]" leader-address= leader-id=
2024-05-07T18:34:33.511Z [INFO]  server-38746261-7d44-fcc9-45f3-8d91eb0a1f74: entering follower state: follower="Node at 38746261-7d44-fcc9-45f3-8d91eb0a1f74 [Follower]" leader-address= leader-id=
2024-05-07T18:34:33.570Z [WARN]  server-38746261-7d44-fcc9-45f3-8d91eb0a1f74: no known peers, aborting election
2024-05-07T18:34:33.578Z [WARN]  server-c810e15b-952b-59bb-350b-b41e8fdb918d: no known peers, aborting election
2024-05-07T18:34:33.610Z [WARN]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:33.610Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: entering candidate state: node="Node at b258852e-dc02-0129-f1a7-850343ca1f4c [Candidate]" term=2
2024-05-07T18:34:33.610Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: election won: term=2 tally=1
2024-05-07T18:34:33.610Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: entering leader state: leader="Node at b258852e-dc02-0129-f1a7-850343ca1f4c [Leader]"
2024-05-07T18:34:33.718Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:33.61069491 +0000 UTC m=+11.211813150 (1 nodes), 99.246914ms from start of poll, 99.56663ms from cluster start. Timeout at 2024-05-07 18:34:33.718490073 +0000 UTC m=+11.319608313, 107.795163ms after stability
2024-05-07T18:34:33.718Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: updating configuration: command=AddVoter server-id=server-c810e15b-952b-59bb-350b-b41e8fdb918d server-addr=c810e15b-952b-59bb-350b-b41e8fdb918d servers="[{Suffrage:Voter ID:server-b258852e-dc02-0129-f1a7-850343ca1f4c Address:b258852e-dc02-0129-f1a7-850343ca1f4c} {Suffrage:Voter ID:server-c810e15b-952b-59bb-350b-b41e8fdb918d Address:c810e15b-952b-59bb-350b-b41e8fdb918d}]"
2024-05-07T18:34:33.718Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: added peer, starting replication: peer=server-c810e15b-952b-59bb-350b-b41e8fdb918d
2024-05-07T18:34:33.718Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:33.718Z [WARN]  server-c810e15b-952b-59bb-350b-b41e8fdb918d: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2024-05-07T18:34:33.718Z [WARN]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: appendEntries rejected, sending older logs: peer="{Voter server-c810e15b-952b-59bb-350b-b41e8fdb918d c810e15b-952b-59bb-350b-b41e8fdb918d}" next=1
2024-05-07T18:34:33.718Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: pipelining replication: peer="{Voter server-c810e15b-952b-59bb-350b-b41e8fdb918d c810e15b-952b-59bb-350b-b41e8fdb918d}"
2024-05-07T18:34:33.826Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:33.718652756 +0000 UTC m=+11.319770996 (1 nodes), 662ns from start of poll, 207.524476ms from cluster start. Timeout at 2024-05-07 18:34:33.826483895 +0000 UTC m=+11.427602145, 107.831149ms after stability
2024-05-07T18:34:33.826Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: updating configuration: command=AddVoter server-id=server-38746261-7d44-fcc9-45f3-8d91eb0a1f74 server-addr=38746261-7d44-fcc9-45f3-8d91eb0a1f74 servers="[{Suffrage:Voter ID:server-b258852e-dc02-0129-f1a7-850343ca1f4c Address:b258852e-dc02-0129-f1a7-850343ca1f4c} {Suffrage:Voter ID:server-c810e15b-952b-59bb-350b-b41e8fdb918d Address:c810e15b-952b-59bb-350b-b41e8fdb918d} {Suffrage:Voter ID:server-38746261-7d44-fcc9-45f3-8d91eb0a1f74 Address:38746261-7d44-fcc9-45f3-8d91eb0a1f74}]"
2024-05-07T18:34:33.826Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: added peer, starting replication: peer=server-38746261-7d44-fcc9-45f3-8d91eb0a1f74
2024-05-07T18:34:33.826Z [WARN]  server-38746261-7d44-fcc9-45f3-8d91eb0a1f74: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2024-05-07T18:34:33.826Z [WARN]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: appendEntries rejected, sending older logs: peer="{Voter server-38746261-7d44-fcc9-45f3-8d91eb0a1f74 38746261-7d44-fcc9-45f3-8d91eb0a1f74}" next=1
2024-05-07T18:34:33.826Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: pipelining replication: peer="{Voter server-38746261-7d44-fcc9-45f3-8d91eb0a1f74 38746261-7d44-fcc9-45f3-8d91eb0a1f74}"
2024-05-07T18:34:33.834Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:33.942Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:33.834671394 +0000 UTC m=+11.435789634 (1 nodes), 1.814µs from start of poll, 323.543114ms from cluster start. Timeout at 2024-05-07 18:34:33.94250089 +0000 UTC m=+11.543619130, 107.829496ms after stability
2024-05-07T18:34:33.942Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: aborting pipeline replication: peer="{Voter server-c810e15b-952b-59bb-350b-b41e8fdb918d c810e15b-952b-59bb-350b-b41e8fdb918d}"
2024-05-07T18:34:33.942Z [INFO]  server-b258852e-dc02-0129-f1a7-850343ca1f4c: aborting pipeline replication: peer="{Voter server-38746261-7d44-fcc9-45f3-8d91eb0a1f74 38746261-7d44-fcc9-45f3-8d91eb0a1f74}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.43s)
=== RUN   TestRaft_RemoveFollower
2024-05-07T18:34:33.943Z [INFO]  server-34ace2f8-f033-6eff-7847-b92d84fbd80e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34ace2f8-f033-6eff-7847-b92d84fbd80e Address:34ace2f8-f033-6eff-7847-b92d84fbd80e} {Suffrage:Voter ID:server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766 Address:b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766} {Suffrage:Voter ID:server-582f7955-2eea-9302-c2b3-632ac2a04158 Address:582f7955-2eea-9302-c2b3-632ac2a04158}]"
2024-05-07T18:34:33.943Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34ace2f8-f033-6eff-7847-b92d84fbd80e Address:34ace2f8-f033-6eff-7847-b92d84fbd80e} {Suffrage:Voter ID:server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766 Address:b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766} {Suffrage:Voter ID:server-582f7955-2eea-9302-c2b3-632ac2a04158 Address:582f7955-2eea-9302-c2b3-632ac2a04158}]"
2024-05-07T18:34:33.943Z [INFO]  server-582f7955-2eea-9302-c2b3-632ac2a04158: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34ace2f8-f033-6eff-7847-b92d84fbd80e Address:34ace2f8-f033-6eff-7847-b92d84fbd80e} {Suffrage:Voter ID:server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766 Address:b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766} {Suffrage:Voter ID:server-582f7955-2eea-9302-c2b3-632ac2a04158 Address:582f7955-2eea-9302-c2b3-632ac2a04158}]"
2024-05-07T18:34:33.943Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:33.943Z [INFO]  server-34ace2f8-f033-6eff-7847-b92d84fbd80e: entering follower state: follower="Node at 34ace2f8-f033-6eff-7847-b92d84fbd80e [Follower]" leader-address= leader-id=
2024-05-07T18:34:33.943Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: entering follower state: follower="Node at b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766 [Follower]" leader-address= leader-id=
2024-05-07T18:34:33.943Z [INFO]  server-582f7955-2eea-9302-c2b3-632ac2a04158: entering follower state: follower="Node at 582f7955-2eea-9302-c2b3-632ac2a04158 [Follower]" leader-address= leader-id=
2024-05-07T18:34:33.998Z [WARN]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:33.998Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: entering candidate state: node="Node at b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766 [Candidate]" term=2
2024-05-07T18:34:33.998Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: election won: term=2 tally=2
2024-05-07T18:34:33.998Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: entering leader state: leader="Node at b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766 [Leader]"
2024-05-07T18:34:33.998Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: added peer, starting replication: peer=server-34ace2f8-f033-6eff-7847-b92d84fbd80e
2024-05-07T18:34:33.998Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: added peer, starting replication: peer=server-582f7955-2eea-9302-c2b3-632ac2a04158
2024-05-07T18:34:33.998Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: pipelining replication: peer="{Voter server-582f7955-2eea-9302-c2b3-632ac2a04158 582f7955-2eea-9302-c2b3-632ac2a04158}"
2024-05-07T18:34:33.998Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: pipelining replication: peer="{Voter server-34ace2f8-f033-6eff-7847-b92d84fbd80e 34ace2f8-f033-6eff-7847-b92d84fbd80e}"
2024-05-07T18:34:34.106Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:33.998915499 +0000 UTC m=+11.600033739 (1 nodes), 55.373983ms from start of poll, 55.569055ms from cluster start. Timeout at 2024-05-07 18:34:34.106492646 +0000 UTC m=+11.707610896, 107.577157ms after stability
2024-05-07T18:34:34.118Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:34.226Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:34.118587211 +0000 UTC m=+11.719705451 (2 nodes), 2.554µs from start of poll, 175.240767ms from cluster start. Timeout at 2024-05-07 18:34:34.22649821 +0000 UTC m=+11.827616460, 107.911009ms after stability
2024-05-07T18:34:34.226Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: updating configuration: command=RemoveServer server-id=server-34ace2f8-f033-6eff-7847-b92d84fbd80e server-addr= servers="[{Suffrage:Voter ID:server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766 Address:b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766} {Suffrage:Voter ID:server-582f7955-2eea-9302-c2b3-632ac2a04158 Address:582f7955-2eea-9302-c2b3-632ac2a04158}]"
2024-05-07T18:34:34.226Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: removed peer, stopping replication: peer=server-34ace2f8-f033-6eff-7847-b92d84fbd80e last-index=3
2024-05-07T18:34:34.226Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: aborting pipeline replication: peer="{Voter server-34ace2f8-f033-6eff-7847-b92d84fbd80e 34ace2f8-f033-6eff-7847-b92d84fbd80e}"
2024-05-07T18:34:34.334Z [INFO]  server-b4b2b9f9-bbe4-52d6-d011-2c9a8fa42766: aborting pipeline replication: peer="{Voter server-582f7955-2eea-9302-c2b3-632ac2a04158 582f7955-2eea-9302-c2b3-632ac2a04158}"
--- PASS: TestRaft_RemoveFollower (0.39s)
=== RUN   TestRaft_RemoveLeader
2024-05-07T18:34:34.335Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3413e792-9b9e-1468-40b7-3073b789e11e Address:3413e792-9b9e-1468-40b7-3073b789e11e} {Suffrage:Voter ID:server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5 Address:aa4dfba3-7fe5-4a36-189c-74f48bb604b5} {Suffrage:Voter ID:server-c6aeac13-2991-a468-ff67-82416f697191 Address:c6aeac13-2991-a468-ff67-82416f697191}]"
2024-05-07T18:34:34.335Z [INFO]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3413e792-9b9e-1468-40b7-3073b789e11e Address:3413e792-9b9e-1468-40b7-3073b789e11e} {Suffrage:Voter ID:server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5 Address:aa4dfba3-7fe5-4a36-189c-74f48bb604b5} {Suffrage:Voter ID:server-c6aeac13-2991-a468-ff67-82416f697191 Address:c6aeac13-2991-a468-ff67-82416f697191}]"
2024-05-07T18:34:34.335Z [INFO]  server-c6aeac13-2991-a468-ff67-82416f697191: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3413e792-9b9e-1468-40b7-3073b789e11e Address:3413e792-9b9e-1468-40b7-3073b789e11e} {Suffrage:Voter ID:server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5 Address:aa4dfba3-7fe5-4a36-189c-74f48bb604b5} {Suffrage:Voter ID:server-c6aeac13-2991-a468-ff67-82416f697191 Address:c6aeac13-2991-a468-ff67-82416f697191}]"
2024-05-07T18:34:34.335Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:34.335Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: entering follower state: follower="Node at 3413e792-9b9e-1468-40b7-3073b789e11e [Follower]" leader-address= leader-id=
2024-05-07T18:34:34.335Z [INFO]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: entering follower state: follower="Node at aa4dfba3-7fe5-4a36-189c-74f48bb604b5 [Follower]" leader-address= leader-id=
2024-05-07T18:34:34.335Z [INFO]  server-c6aeac13-2991-a468-ff67-82416f697191: entering follower state: follower="Node at c6aeac13-2991-a468-ff67-82416f697191 [Follower]" leader-address= leader-id=
2024-05-07T18:34:34.387Z [WARN]  server-3413e792-9b9e-1468-40b7-3073b789e11e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:34.387Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: entering candidate state: node="Node at 3413e792-9b9e-1468-40b7-3073b789e11e [Candidate]" term=2
2024-05-07T18:34:34.388Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: election won: term=2 tally=2
2024-05-07T18:34:34.388Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: entering leader state: leader="Node at 3413e792-9b9e-1468-40b7-3073b789e11e [Leader]"
2024-05-07T18:34:34.388Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: added peer, starting replication: peer=server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5
2024-05-07T18:34:34.388Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: added peer, starting replication: peer=server-c6aeac13-2991-a468-ff67-82416f697191
2024-05-07T18:34:34.388Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: pipelining replication: peer="{Voter server-c6aeac13-2991-a468-ff67-82416f697191 c6aeac13-2991-a468-ff67-82416f697191}"
2024-05-07T18:34:34.388Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: pipelining replication: peer="{Voter server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5 aa4dfba3-7fe5-4a36-189c-74f48bb604b5}"
2024-05-07T18:34:34.494Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:34.388253747 +0000 UTC m=+11.989371987 (1 nodes), 52.663451ms from start of poll, 52.865026ms from cluster start. Timeout at 2024-05-07 18:34:34.494490231 +0000 UTC m=+12.095608471, 106.236484ms after stability
2024-05-07T18:34:34.502Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:34.614Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:34.502561543 +0000 UTC m=+12.103679784 (2 nodes), 1.413µs from start of poll, 167.172823ms from cluster start. Timeout at 2024-05-07 18:34:34.614481869 +0000 UTC m=+12.215600109, 111.920325ms after stability
2024-05-07T18:34:34.614Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: updating configuration: command=RemoveServer server-id=server-3413e792-9b9e-1468-40b7-3073b789e11e server-addr= servers="[{Suffrage:Voter ID:server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5 Address:aa4dfba3-7fe5-4a36-189c-74f48bb604b5} {Suffrage:Voter ID:server-c6aeac13-2991-a468-ff67-82416f697191 Address:c6aeac13-2991-a468-ff67-82416f697191}]"
2024-05-07T18:34:34.614Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: removed ourself, shutting down
2024-05-07T18:34:34.614Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: aborting pipeline replication: peer="{Voter server-c6aeac13-2991-a468-ff67-82416f697191 c6aeac13-2991-a468-ff67-82416f697191}"
2024-05-07T18:34:34.614Z [INFO]  server-3413e792-9b9e-1468-40b7-3073b789e11e: aborting pipeline replication: peer="{Voter server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5 aa4dfba3-7fe5-4a36-189c-74f48bb604b5}"
2024-05-07T18:34:34.690Z [WARN]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: heartbeat timeout reached, starting election: last-leader-addr=3413e792-9b9e-1468-40b7-3073b789e11e last-leader-id=server-3413e792-9b9e-1468-40b7-3073b789e11e
2024-05-07T18:34:34.690Z [INFO]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: entering candidate state: node="Node at aa4dfba3-7fe5-4a36-189c-74f48bb604b5 [Candidate]" term=3
2024-05-07T18:34:34.690Z [WARN]  server-c6aeac13-2991-a468-ff67-82416f697191: rejecting vote request since we have a leader: from=aa4dfba3-7fe5-4a36-189c-74f48bb604b5 leader=3413e792-9b9e-1468-40b7-3073b789e11e leader-id=server-3413e792-9b9e-1468-40b7-3073b789e11e
2024-05-07T18:34:34.714Z [WARN]  server-c6aeac13-2991-a468-ff67-82416f697191: heartbeat timeout reached, starting election: last-leader-addr=3413e792-9b9e-1468-40b7-3073b789e11e last-leader-id=server-3413e792-9b9e-1468-40b7-3073b789e11e
2024-05-07T18:34:34.714Z [INFO]  server-c6aeac13-2991-a468-ff67-82416f697191: entering candidate state: node="Node at c6aeac13-2991-a468-ff67-82416f697191 [Candidate]" term=3
2024-05-07T18:34:34.714Z [INFO]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: duplicate requestVote for same term: term=3
2024-05-07T18:34:34.766Z [WARN]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: Election timeout reached, restarting election
2024-05-07T18:34:34.766Z [INFO]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: entering candidate state: node="Node at aa4dfba3-7fe5-4a36-189c-74f48bb604b5 [Candidate]" term=4
2024-05-07T18:34:34.766Z [INFO]  server-c6aeac13-2991-a468-ff67-82416f697191: entering follower state: follower="Node at c6aeac13-2991-a468-ff67-82416f697191 [Follower]" leader-address= leader-id=
2024-05-07T18:34:34.766Z [INFO]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: election won: term=4 tally=2
2024-05-07T18:34:34.766Z [INFO]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: entering leader state: leader="Node at aa4dfba3-7fe5-4a36-189c-74f48bb604b5 [Leader]"
2024-05-07T18:34:34.766Z [INFO]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: added peer, starting replication: peer=server-c6aeac13-2991-a468-ff67-82416f697191
2024-05-07T18:34:34.766Z [INFO]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: pipelining replication: peer="{Voter server-c6aeac13-2991-a468-ff67-82416f697191 c6aeac13-2991-a468-ff67-82416f697191}"
2024-05-07T18:34:34.830Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:34.946Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:34.838546998 +0000 UTC m=+12.439665238 (1 nodes), 8.001182ms from start of poll, 503.158277ms from cluster start. Timeout at 2024-05-07 18:34:34.946547683 +0000 UTC m=+12.547665923, 108.000685ms after stability
2024-05-07T18:34:34.946Z [INFO]  server-aa4dfba3-7fe5-4a36-189c-74f48bb604b5: aborting pipeline replication: peer="{Voter server-c6aeac13-2991-a468-ff67-82416f697191 c6aeac13-2991-a468-ff67-82416f697191}"
--- PASS: TestRaft_RemoveLeader (0.61s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2024-05-07T18:34:34.947Z [INFO]  server-f7f6239e-0947-5734-964f-17a63a85dd5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7f6239e-0947-5734-964f-17a63a85dd5f Address:f7f6239e-0947-5734-964f-17a63a85dd5f} {Suffrage:Voter ID:server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a Address:94e3c307-1fd3-c5ca-1eb9-f33422a7e40a} {Suffrage:Voter ID:server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7 Address:f8338489-f8f7-1de3-1cd4-c70b7130d8f7}]"
2024-05-07T18:34:34.947Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7f6239e-0947-5734-964f-17a63a85dd5f Address:f7f6239e-0947-5734-964f-17a63a85dd5f} {Suffrage:Voter ID:server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a Address:94e3c307-1fd3-c5ca-1eb9-f33422a7e40a} {Suffrage:Voter ID:server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7 Address:f8338489-f8f7-1de3-1cd4-c70b7130d8f7}]"
2024-05-07T18:34:34.947Z [INFO]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7f6239e-0947-5734-964f-17a63a85dd5f Address:f7f6239e-0947-5734-964f-17a63a85dd5f} {Suffrage:Voter ID:server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a Address:94e3c307-1fd3-c5ca-1eb9-f33422a7e40a} {Suffrage:Voter ID:server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7 Address:f8338489-f8f7-1de3-1cd4-c70b7130d8f7}]"
2024-05-07T18:34:34.947Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:34.947Z [INFO]  server-f7f6239e-0947-5734-964f-17a63a85dd5f: entering follower state: follower="Node at f7f6239e-0947-5734-964f-17a63a85dd5f [Follower]" leader-address= leader-id=
2024-05-07T18:34:34.947Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: entering follower state: follower="Node at 94e3c307-1fd3-c5ca-1eb9-f33422a7e40a [Follower]" leader-address= leader-id=
2024-05-07T18:34:34.947Z [INFO]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: entering follower state: follower="Node at f8338489-f8f7-1de3-1cd4-c70b7130d8f7 [Follower]" leader-address= leader-id=
2024-05-07T18:34:35.010Z [WARN]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:35.010Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: entering candidate state: node="Node at 94e3c307-1fd3-c5ca-1eb9-f33422a7e40a [Candidate]" term=2
2024-05-07T18:34:35.010Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: election won: term=2 tally=2
2024-05-07T18:34:35.010Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: entering leader state: leader="Node at 94e3c307-1fd3-c5ca-1eb9-f33422a7e40a [Leader]"
2024-05-07T18:34:35.010Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: added peer, starting replication: peer=server-f7f6239e-0947-5734-964f-17a63a85dd5f
2024-05-07T18:34:35.010Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: added peer, starting replication: peer=server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7
2024-05-07T18:34:35.010Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: pipelining replication: peer="{Voter server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7 f8338489-f8f7-1de3-1cd4-c70b7130d8f7}"
2024-05-07T18:34:35.010Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: pipelining replication: peer="{Voter server-f7f6239e-0947-5734-964f-17a63a85dd5f f7f6239e-0947-5734-964f-17a63a85dd5f}"
2024-05-07T18:34:35.118Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:35.010851385 +0000 UTC m=+12.611969625 (2 nodes), 63.363643ms from start of poll, 63.559828ms from cluster start. Timeout at 2024-05-07 18:34:35.118549988 +0000 UTC m=+12.719668228, 107.698603ms after stability
2024-05-07T18:34:35.118Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:35.226Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:35.118607125 +0000 UTC m=+12.719725365 (1 nodes), 671ns from start of poll, 171.315568ms from cluster start. Timeout at 2024-05-07 18:34:35.226486304 +0000 UTC m=+12.827604544, 107.879179ms after stability
2024-05-07T18:34:35.228Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: updating configuration: command=RemoveServer server-id=server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a server-addr= servers="[{Suffrage:Voter ID:server-f7f6239e-0947-5734-964f-17a63a85dd5f Address:f7f6239e-0947-5734-964f-17a63a85dd5f} {Suffrage:Voter ID:server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7 Address:f8338489-f8f7-1de3-1cd4-c70b7130d8f7}]"
2024-05-07T18:34:35.228Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: removed ourself, transitioning to follower
2024-05-07T18:34:35.228Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: entering follower state: follower="Node at 94e3c307-1fd3-c5ca-1eb9-f33422a7e40a [Follower]" leader-address= leader-id=
2024-05-07T18:34:35.228Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: aborting pipeline replication: peer="{Voter server-f7f6239e-0947-5734-964f-17a63a85dd5f f7f6239e-0947-5734-964f-17a63a85dd5f}"
2024-05-07T18:34:35.229Z [INFO]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: aborting pipeline replication: peer="{Voter server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7 f8338489-f8f7-1de3-1cd4-c70b7130d8f7}"
2024-05-07T18:34:35.318Z [WARN]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: heartbeat timeout reached, starting election: last-leader-addr=94e3c307-1fd3-c5ca-1eb9-f33422a7e40a last-leader-id=server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a
2024-05-07T18:34:35.318Z [INFO]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: entering candidate state: node="Node at f8338489-f8f7-1de3-1cd4-c70b7130d8f7 [Candidate]" term=3
2024-05-07T18:34:35.318Z [WARN]  server-f7f6239e-0947-5734-964f-17a63a85dd5f: rejecting vote request since we have a leader: from=f8338489-f8f7-1de3-1cd4-c70b7130d8f7 leader=94e3c307-1fd3-c5ca-1eb9-f33422a7e40a leader-id=server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a
2024-05-07T18:34:35.330Z [WARN]  server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a: not part of stable configuration, aborting election
2024-05-07T18:34:35.334Z [WARN]  server-f7f6239e-0947-5734-964f-17a63a85dd5f: heartbeat timeout reached, starting election: last-leader-addr=94e3c307-1fd3-c5ca-1eb9-f33422a7e40a last-leader-id=server-94e3c307-1fd3-c5ca-1eb9-f33422a7e40a
2024-05-07T18:34:35.334Z [INFO]  server-f7f6239e-0947-5734-964f-17a63a85dd5f: entering candidate state: node="Node at f7f6239e-0947-5734-964f-17a63a85dd5f [Candidate]" term=3
2024-05-07T18:34:35.334Z [INFO]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: duplicate requestVote for same term: term=3
2024-05-07T18:34:35.342Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:35.378Z [WARN]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: Election timeout reached, restarting election
2024-05-07T18:34:35.378Z [INFO]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: entering candidate state: node="Node at f8338489-f8f7-1de3-1cd4-c70b7130d8f7 [Candidate]" term=4
2024-05-07T18:34:35.378Z [INFO]  server-f7f6239e-0947-5734-964f-17a63a85dd5f: entering follower state: follower="Node at f7f6239e-0947-5734-964f-17a63a85dd5f [Follower]" leader-address= leader-id=
2024-05-07T18:34:35.378Z [INFO]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: election won: term=4 tally=2
2024-05-07T18:34:35.378Z [INFO]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: entering leader state: leader="Node at f8338489-f8f7-1de3-1cd4-c70b7130d8f7 [Leader]"
2024-05-07T18:34:35.378Z [INFO]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: added peer, starting replication: peer=server-f7f6239e-0947-5734-964f-17a63a85dd5f
2024-05-07T18:34:35.378Z [INFO]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: pipelining replication: peer="{Voter server-f7f6239e-0947-5734-964f-17a63a85dd5f f7f6239e-0947-5734-964f-17a63a85dd5f}"
2024-05-07T18:34:35.486Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:35.378759304 +0000 UTC m=+12.979877544 (1 nodes), 36.217233ms from start of poll, 431.467747ms from cluster start. Timeout at 2024-05-07 18:34:35.486503192 +0000 UTC m=+13.087621432, 107.743888ms after stability
2024-05-07T18:34:35.594Z [INFO]  server-f8338489-f8f7-1de3-1cd4-c70b7130d8f7: aborting pipeline replication: peer="{Voter server-f7f6239e-0947-5734-964f-17a63a85dd5f f7f6239e-0947-5734-964f-17a63a85dd5f}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.65s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2024-05-07T18:34:35.595Z [INFO]  server-282cade4-2f13-4d5f-c51e-d3d195586567: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-282cade4-2f13-4d5f-c51e-d3d195586567 Address:282cade4-2f13-4d5f-c51e-d3d195586567} {Suffrage:Voter ID:server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 Address:c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6} {Suffrage:Voter ID:server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63 Address:ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63} {Suffrage:Voter ID:server-4a6d66d2-c5c5-e71f-3d50-e75108aac171 Address:4a6d66d2-c5c5-e71f-3d50-e75108aac171}]"
2024-05-07T18:34:35.595Z [INFO]  server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-282cade4-2f13-4d5f-c51e-d3d195586567 Address:282cade4-2f13-4d5f-c51e-d3d195586567} {Suffrage:Voter ID:server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 Address:c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6} {Suffrage:Voter ID:server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63 Address:ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63} {Suffrage:Voter ID:server-4a6d66d2-c5c5-e71f-3d50-e75108aac171 Address:4a6d66d2-c5c5-e71f-3d50-e75108aac171}]"
2024-05-07T18:34:35.595Z [INFO]  server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-282cade4-2f13-4d5f-c51e-d3d195586567 Address:282cade4-2f13-4d5f-c51e-d3d195586567} {Suffrage:Voter ID:server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 Address:c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6} {Suffrage:Voter ID:server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63 Address:ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63} {Suffrage:Voter ID:server-4a6d66d2-c5c5-e71f-3d50-e75108aac171 Address:4a6d66d2-c5c5-e71f-3d50-e75108aac171}]"
2024-05-07T18:34:35.595Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-282cade4-2f13-4d5f-c51e-d3d195586567 Address:282cade4-2f13-4d5f-c51e-d3d195586567} {Suffrage:Voter ID:server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 Address:c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6} {Suffrage:Voter ID:server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63 Address:ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63} {Suffrage:Voter ID:server-4a6d66d2-c5c5-e71f-3d50-e75108aac171 Address:4a6d66d2-c5c5-e71f-3d50-e75108aac171}]"
2024-05-07T18:34:35.595Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:35.595Z [INFO]  server-282cade4-2f13-4d5f-c51e-d3d195586567: entering follower state: follower="Node at 282cade4-2f13-4d5f-c51e-d3d195586567 [Follower]" leader-address= leader-id=
2024-05-07T18:34:35.595Z [INFO]  server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6: entering follower state: follower="Node at c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 [Follower]" leader-address= leader-id=
2024-05-07T18:34:35.595Z [INFO]  server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63: entering follower state: follower="Node at ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63 [Follower]" leader-address= leader-id=
2024-05-07T18:34:35.595Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: entering follower state: follower="Node at 4a6d66d2-c5c5-e71f-3d50-e75108aac171 [Follower]" leader-address= leader-id=
2024-05-07T18:34:35.662Z [WARN]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:35.662Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: entering candidate state: node="Node at 4a6d66d2-c5c5-e71f-3d50-e75108aac171 [Candidate]" term=2
2024-05-07T18:34:35.662Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: election won: term=2 tally=3
2024-05-07T18:34:35.662Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: entering leader state: leader="Node at 4a6d66d2-c5c5-e71f-3d50-e75108aac171 [Leader]"
2024-05-07T18:34:35.662Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: added peer, starting replication: peer=server-282cade4-2f13-4d5f-c51e-d3d195586567
2024-05-07T18:34:35.662Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: added peer, starting replication: peer=server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6
2024-05-07T18:34:35.662Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: added peer, starting replication: peer=server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63
2024-05-07T18:34:35.662Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: pipelining replication: peer="{Voter server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63 ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63}"
2024-05-07T18:34:35.662Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: pipelining replication: peer="{Voter server-282cade4-2f13-4d5f-c51e-d3d195586567 282cade4-2f13-4d5f-c51e-d3d195586567}"
2024-05-07T18:34:35.662Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: pipelining replication: peer="{Voter server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6}"
2024-05-07T18:34:35.770Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:35.662806717 +0000 UTC m=+13.263924957 (1 nodes), 67.215659ms from start of poll, 67.450084ms from cluster start. Timeout at 2024-05-07 18:34:35.7704996 +0000 UTC m=+13.371617860, 107.692903ms after stability
2024-05-07T18:34:35.878Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:35.984Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:35.878584913 +0000 UTC m=+13.479703173 (3 nodes), 1.904µs from start of poll, 283.2283ms from cluster start. Timeout at 2024-05-07 18:34:35.984391828 +0000 UTC m=+13.585510068, 105.806895ms after stability
2024-05-07T18:34:35.984Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: updating configuration: command=RemoveServer server-id=server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63 server-addr= servers="[{Suffrage:Voter ID:server-282cade4-2f13-4d5f-c51e-d3d195586567 Address:282cade4-2f13-4d5f-c51e-d3d195586567} {Suffrage:Voter ID:server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 Address:c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6} {Suffrage:Voter ID:server-4a6d66d2-c5c5-e71f-3d50-e75108aac171 Address:4a6d66d2-c5c5-e71f-3d50-e75108aac171}]"
2024-05-07T18:34:35.984Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: removed peer, stopping replication: peer=server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63 last-index=3
2024-05-07T18:34:35.984Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: aborting pipeline replication: peer="{Voter server-ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63 ed7a5bea-ca83-ebf1-cf60-4bc6ecdb9e63}"
2024-05-07T18:34:35.984Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to pipeline appendEntries: peer="{Voter server-282cade4-2f13-4d5f-c51e-d3d195586567 282cade4-2f13-4d5f-c51e-d3d195586567}" error="append pipeline closed"
2024-05-07T18:34:35.984Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: aborting pipeline replication: peer="{Voter server-282cade4-2f13-4d5f-c51e-d3d195586567 282cade4-2f13-4d5f-c51e-d3d195586567}"
2024-05-07T18:34:35.984Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to pipeline appendEntries: peer="{Voter server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6}" error="append pipeline closed"
2024-05-07T18:34:35.984Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: aborting pipeline replication: peer="{Voter server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6}"
2024-05-07T18:34:35.989Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to heartbeat to: peer=282cade4-2f13-4d5f-c51e-d3d195586567 backoff time=10ms error="failed to connect to peer: 282cade4-2f13-4d5f-c51e-d3d195586567"
2024-05-07T18:34:35.990Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to appendEntries to: peer="{Voter server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6}" error="failed to connect to peer: c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6"
2024-05-07T18:34:35.992Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to heartbeat to: peer=c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 backoff time=10ms error="failed to connect to peer: c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6"
2024-05-07T18:34:35.992Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to appendEntries to: peer="{Voter server-282cade4-2f13-4d5f-c51e-d3d195586567 282cade4-2f13-4d5f-c51e-d3d195586567}" error="failed to connect to peer: 282cade4-2f13-4d5f-c51e-d3d195586567"
2024-05-07T18:34:36.008Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to appendEntries to: peer="{Voter server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6}" error="failed to connect to peer: c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6"
2024-05-07T18:34:36.008Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to heartbeat to: peer=282cade4-2f13-4d5f-c51e-d3d195586567 backoff time=10ms error="failed to connect to peer: 282cade4-2f13-4d5f-c51e-d3d195586567"
2024-05-07T18:34:36.009Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to appendEntries to: peer="{Voter server-282cade4-2f13-4d5f-c51e-d3d195586567 282cade4-2f13-4d5f-c51e-d3d195586567}" error="failed to connect to peer: 282cade4-2f13-4d5f-c51e-d3d195586567"
2024-05-07T18:34:36.012Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to heartbeat to: peer=c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 backoff time=10ms error="failed to connect to peer: c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6"
2024-05-07T18:34:36.026Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to appendEntries to: peer="{Voter server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6}" error="failed to connect to peer: c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6"
2024-05-07T18:34:36.026Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to appendEntries to: peer="{Voter server-282cade4-2f13-4d5f-c51e-d3d195586567 282cade4-2f13-4d5f-c51e-d3d195586567}" error="failed to connect to peer: 282cade4-2f13-4d5f-c51e-d3d195586567"
2024-05-07T18:34:36.028Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to heartbeat to: peer=282cade4-2f13-4d5f-c51e-d3d195586567 backoff time=10ms error="failed to connect to peer: 282cade4-2f13-4d5f-c51e-d3d195586567"
2024-05-07T18:34:36.030Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to heartbeat to: peer=c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 backoff time=10ms error="failed to connect to peer: c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6"
2024-05-07T18:34:36.035Z [WARN]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to contact: server-id=server-282cade4-2f13-4d5f-c51e-d3d195586567 time=54.459782ms
2024-05-07T18:34:36.035Z [WARN]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to contact: server-id=server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 time=52.129167ms
2024-05-07T18:34:36.035Z [WARN]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to contact quorum of nodes, stepping down
2024-05-07T18:34:36.035Z [INFO]  server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: entering follower state: follower="Node at 4a6d66d2-c5c5-e71f-3d50-e75108aac171 [Follower]" leader-address= leader-id=
2024-05-07T18:34:36.035Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to appendEntries to: peer="{Voter server-c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6 c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6}" error="failed to connect to peer: c88e8f7c-b21c-6d63-33dd-4d465e4fe4f6"
2024-05-07T18:34:36.035Z [ERROR] server-4a6d66d2-c5c5-e71f-3d50-e75108aac171: failed to appendEntries to: peer="{Voter server-282cade4-2f13-4d5f-c51e-d3d195586567 282cade4-2f13-4d5f-c51e-d3d195586567}" error="failed to connect to peer: 282cade4-2f13-4d5f-c51e-d3d195586567"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s)
=== RUN   TestRaft_AddKnownPeer
2024-05-07T18:34:36.036Z [INFO]  server-a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8 Address:a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8} {Suffrage:Voter ID:server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4 Address:ddbcd1c9-8164-10e4-c320-2b6ebee837f4} {Suffrage:Voter ID:server-1d5b40c5-f447-e57a-cac9-d26773f71709 Address:1d5b40c5-f447-e57a-cac9-d26773f71709}]"
2024-05-07T18:34:36.036Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8 Address:a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8} {Suffrage:Voter ID:server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4 Address:ddbcd1c9-8164-10e4-c320-2b6ebee837f4} {Suffrage:Voter ID:server-1d5b40c5-f447-e57a-cac9-d26773f71709 Address:1d5b40c5-f447-e57a-cac9-d26773f71709}]"
2024-05-07T18:34:36.036Z [INFO]  server-1d5b40c5-f447-e57a-cac9-d26773f71709: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8 Address:a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8} {Suffrage:Voter ID:server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4 Address:ddbcd1c9-8164-10e4-c320-2b6ebee837f4} {Suffrage:Voter ID:server-1d5b40c5-f447-e57a-cac9-d26773f71709 Address:1d5b40c5-f447-e57a-cac9-d26773f71709}]"
2024-05-07T18:34:36.036Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:36.037Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: entering follower state: follower="Node at ddbcd1c9-8164-10e4-c320-2b6ebee837f4 [Follower]" leader-address= leader-id=
2024-05-07T18:34:36.037Z [INFO]  server-a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8: entering follower state: follower="Node at a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8 [Follower]" leader-address= leader-id=
2024-05-07T18:34:36.037Z [INFO]  server-1d5b40c5-f447-e57a-cac9-d26773f71709: entering follower state: follower="Node at 1d5b40c5-f447-e57a-cac9-d26773f71709 [Follower]" leader-address= leader-id=
2024-05-07T18:34:36.113Z [WARN]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:36.113Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: entering candidate state: node="Node at ddbcd1c9-8164-10e4-c320-2b6ebee837f4 [Candidate]" term=2
2024-05-07T18:34:36.113Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: election won: term=2 tally=2
2024-05-07T18:34:36.113Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: entering leader state: leader="Node at ddbcd1c9-8164-10e4-c320-2b6ebee837f4 [Leader]"
2024-05-07T18:34:36.113Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: added peer, starting replication: peer=server-a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8
2024-05-07T18:34:36.113Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: added peer, starting replication: peer=server-1d5b40c5-f447-e57a-cac9-d26773f71709
2024-05-07T18:34:36.114Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: pipelining replication: peer="{Voter server-1d5b40c5-f447-e57a-cac9-d26773f71709 1d5b40c5-f447-e57a-cac9-d26773f71709}"
2024-05-07T18:34:36.114Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: pipelining replication: peer="{Voter server-a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8 a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8}"
2024-05-07T18:34:36.222Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:36.114157083 +0000 UTC m=+13.715275323 (1 nodes), 77.272552ms from start of poll, 77.600191ms from cluster start. Timeout at 2024-05-07 18:34:36.222523329 +0000 UTC m=+13.823641569, 108.366246ms after stability
2024-05-07T18:34:36.222Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:36.330Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:36.222607225 +0000 UTC m=+13.823725465 (2 nodes), 2.965µs from start of poll, 186.050333ms from cluster start. Timeout at 2024-05-07 18:34:36.330476646 +0000 UTC m=+13.931594886, 107.869421ms after stability
2024-05-07T18:34:36.334Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: updating configuration: command=AddVoter server-id=server-a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8 server-addr=a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8 servers="[{Suffrage:Voter ID:server-a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8 Address:a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8} {Suffrage:Voter ID:server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4 Address:ddbcd1c9-8164-10e4-c320-2b6ebee837f4} {Suffrage:Voter ID:server-1d5b40c5-f447-e57a-cac9-d26773f71709 Address:1d5b40c5-f447-e57a-cac9-d26773f71709}]"
2024-05-07T18:34:36.334Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: aborting pipeline replication: peer="{Voter server-a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8 a921f5a8-43a7-d4f9-ddbe-148ab19f1fd8}"
2024-05-07T18:34:36.334Z [INFO]  server-ddbcd1c9-8164-10e4-c320-2b6ebee837f4: aborting pipeline replication: peer="{Voter server-1d5b40c5-f447-e57a-cac9-d26773f71709 1d5b40c5-f447-e57a-cac9-d26773f71709}"
--- PASS: TestRaft_AddKnownPeer (0.30s)
=== RUN   TestRaft_RemoveUnknownPeer
2024-05-07T18:34:36.335Z [INFO]  server-d08ff36a-743f-6396-029f-9f158ab3cdc6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d08ff36a-743f-6396-029f-9f158ab3cdc6 Address:d08ff36a-743f-6396-029f-9f158ab3cdc6} {Suffrage:Voter ID:server-dc15fe65-39e4-0de1-ac80-223d299c6a93 Address:dc15fe65-39e4-0de1-ac80-223d299c6a93} {Suffrage:Voter ID:server-12afc053-a77f-78d4-e3ef-cc22b9d13c40 Address:12afc053-a77f-78d4-e3ef-cc22b9d13c40}]"
2024-05-07T18:34:36.335Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d08ff36a-743f-6396-029f-9f158ab3cdc6 Address:d08ff36a-743f-6396-029f-9f158ab3cdc6} {Suffrage:Voter ID:server-dc15fe65-39e4-0de1-ac80-223d299c6a93 Address:dc15fe65-39e4-0de1-ac80-223d299c6a93} {Suffrage:Voter ID:server-12afc053-a77f-78d4-e3ef-cc22b9d13c40 Address:12afc053-a77f-78d4-e3ef-cc22b9d13c40}]"
2024-05-07T18:34:36.335Z [INFO]  server-12afc053-a77f-78d4-e3ef-cc22b9d13c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d08ff36a-743f-6396-029f-9f158ab3cdc6 Address:d08ff36a-743f-6396-029f-9f158ab3cdc6} {Suffrage:Voter ID:server-dc15fe65-39e4-0de1-ac80-223d299c6a93 Address:dc15fe65-39e4-0de1-ac80-223d299c6a93} {Suffrage:Voter ID:server-12afc053-a77f-78d4-e3ef-cc22b9d13c40 Address:12afc053-a77f-78d4-e3ef-cc22b9d13c40}]"
2024-05-07T18:34:36.335Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:36.335Z [INFO]  server-d08ff36a-743f-6396-029f-9f158ab3cdc6: entering follower state: follower="Node at d08ff36a-743f-6396-029f-9f158ab3cdc6 [Follower]" leader-address= leader-id=
2024-05-07T18:34:36.335Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: entering follower state: follower="Node at dc15fe65-39e4-0de1-ac80-223d299c6a93 [Follower]" leader-address= leader-id=
2024-05-07T18:34:36.335Z [INFO]  server-12afc053-a77f-78d4-e3ef-cc22b9d13c40: entering follower state: follower="Node at 12afc053-a77f-78d4-e3ef-cc22b9d13c40 [Follower]" leader-address= leader-id=
2024-05-07T18:34:36.390Z [WARN]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:36.390Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: entering candidate state: node="Node at dc15fe65-39e4-0de1-ac80-223d299c6a93 [Candidate]" term=2
2024-05-07T18:34:36.390Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: election won: term=2 tally=2
2024-05-07T18:34:36.390Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: entering leader state: leader="Node at dc15fe65-39e4-0de1-ac80-223d299c6a93 [Leader]"
2024-05-07T18:34:36.390Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: added peer, starting replication: peer=server-d08ff36a-743f-6396-029f-9f158ab3cdc6
2024-05-07T18:34:36.390Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: added peer, starting replication: peer=server-12afc053-a77f-78d4-e3ef-cc22b9d13c40
2024-05-07T18:34:36.390Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: pipelining replication: peer="{Voter server-12afc053-a77f-78d4-e3ef-cc22b9d13c40 12afc053-a77f-78d4-e3ef-cc22b9d13c40}"
2024-05-07T18:34:36.390Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: pipelining replication: peer="{Voter server-d08ff36a-743f-6396-029f-9f158ab3cdc6 d08ff36a-743f-6396-029f-9f158ab3cdc6}"
2024-05-07T18:34:36.498Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:36.390926369 +0000 UTC m=+13.992044609 (1 nodes), 55.221539ms from start of poll, 55.395732ms from cluster start. Timeout at 2024-05-07 18:34:36.498491395 +0000 UTC m=+14.099609635, 107.565026ms after stability
2024-05-07T18:34:36.498Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: updating configuration: command=RemoveServer server-id=06524ecb-471a-b9ef-7a03-f0a777d8c5b8 server-addr= servers="[{Suffrage:Voter ID:server-d08ff36a-743f-6396-029f-9f158ab3cdc6 Address:d08ff36a-743f-6396-029f-9f158ab3cdc6} {Suffrage:Voter ID:server-dc15fe65-39e4-0de1-ac80-223d299c6a93 Address:dc15fe65-39e4-0de1-ac80-223d299c6a93} {Suffrage:Voter ID:server-12afc053-a77f-78d4-e3ef-cc22b9d13c40 Address:12afc053-a77f-78d4-e3ef-cc22b9d13c40}]"
2024-05-07T18:34:36.498Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: aborting pipeline replication: peer="{Voter server-12afc053-a77f-78d4-e3ef-cc22b9d13c40 12afc053-a77f-78d4-e3ef-cc22b9d13c40}"
2024-05-07T18:34:36.498Z [INFO]  server-dc15fe65-39e4-0de1-ac80-223d299c6a93: aborting pipeline replication: peer="{Voter server-d08ff36a-743f-6396-029f-9f158ab3cdc6 d08ff36a-743f-6396-029f-9f158ab3cdc6}"
--- PASS: TestRaft_RemoveUnknownPeer (0.16s)
=== RUN   TestRaft_SnapshotRestore
2024-05-07T18:34:36.499Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33e18b7d-b818-9601-fc20-27605f003338 Address:33e18b7d-b818-9601-fc20-27605f003338}]"
2024-05-07T18:34:36.499Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:36.499Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: entering follower state: follower="Node at 33e18b7d-b818-9601-fc20-27605f003338 [Follower]" leader-address= leader-id=
2024-05-07T18:34:36.586Z [WARN]  server-33e18b7d-b818-9601-fc20-27605f003338: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:36.586Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: entering candidate state: node="Node at 33e18b7d-b818-9601-fc20-27605f003338 [Candidate]" term=2
2024-05-07T18:34:36.586Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: election won: term=2 tally=1
2024-05-07T18:34:36.586Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: entering leader state: leader="Node at 33e18b7d-b818-9601-fc20-27605f003338 [Leader]"
2024-05-07T18:34:36.697Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:36.586665273 +0000 UTC m=+14.187783513 (1 nodes), 87.049517ms from start of poll, 87.151266ms from cluster start. Timeout at 2024-05-07 18:34:36.697559942 +0000 UTC m=+14.298678182, 110.894669ms after stability
2024-05-07T18:34:36.698Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: starting snapshot up to: index=102
2024-05-07T18:34:36.698Z [INFO]  creating new snapshot: path=/tmp/raft2110840207/snapshots/2-102-1715106876698.tmp
2024-05-07T18:34:36.698Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: compacting logs: from=1 to=92
2024-05-07T18:34:36.698Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: snapshot complete up to: index=102
2024-05-07T18:34:36.699Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: starting restore from snapshot: id=2-102-1715106876698 last-index=102 last-term=2 size-in-bytes=693
2024-05-07T18:34:36.699Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: snapshot restore progress: id=2-102-1715106876698 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2024-05-07T18:34:36.699Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: restored from snapshot: id=2-102-1715106876698 last-index=102 last-term=2 size-in-bytes=693
2024-05-07T18:34:36.699Z [INFO]  server-33e18b7d-b818-9601-fc20-27605f003338: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33e18b7d-b818-9601-fc20-27605f003338 Address:33e18b7d-b818-9601-fc20-27605f003338}]"
--- PASS: TestRaft_SnapshotRestore (0.20s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2024-05-07T18:34:36.702Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b6829f5-c565-46d8-3fd2-10feb582e087 Address:4b6829f5-c565-46d8-3fd2-10feb582e087}]"
2024-05-07T18:34:36.702Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:36.702Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: entering follower state: follower="Node at 4b6829f5-c565-46d8-3fd2-10feb582e087 [Follower]" leader-address= leader-id=
2024-05-07T18:34:36.786Z [WARN]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:36.786Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: entering candidate state: node="Node at 4b6829f5-c565-46d8-3fd2-10feb582e087 [Candidate]" term=2
2024-05-07T18:34:36.786Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: election won: term=2 tally=1
2024-05-07T18:34:36.786Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: entering leader state: leader="Node at 4b6829f5-c565-46d8-3fd2-10feb582e087 [Leader]"
2024-05-07T18:34:36.892Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:36.786756802 +0000 UTC m=+14.387875042 (1 nodes), 83.778773ms from start of poll, 83.859173ms from cluster start. Timeout at 2024-05-07 18:34:36.892156912 +0000 UTC m=+14.493275152, 105.40011ms after stability
2024-05-07T18:34:36.893Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: starting snapshot up to: index=102
2024-05-07T18:34:36.893Z [INFO]  creating new snapshot: path=/tmp/raft1149870159/snapshots/2-102-1715106876893.tmp
2024-05-07T18:34:36.893Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: compacting logs: from=1 to=92
2024-05-07T18:34:36.893Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: snapshot complete up to: index=102
2024-05-07T18:34:36.894Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: starting restore from snapshot: id=2-102-1715106876893 last-index=102 last-term=2 size-in-bytes=693
2024-05-07T18:34:36.895Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: snapshot restore progress: id=2-102-1715106876893 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2024-05-07T18:34:36.895Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: restored from snapshot: id=2-102-1715106876893 last-index=102 last-term=2 size-in-bytes=693
2024-05-07T18:34:36.895Z [INFO]  server-4b6829f5-c565-46d8-3fd2-10feb582e087: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b6829f5-c565-46d8-3fd2-10feb582e087 Address:4b6829f5-c565-46d8-3fd2-10feb582e087}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.19s)
=== RUN   TestRaft_SnapshotRestore_Progress
2024-05-07T18:34:36.895Z [INFO]  server-89c13de8-7870-23a8-398c-15e6e83704ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89c13de8-7870-23a8-398c-15e6e83704ee Address:89c13de8-7870-23a8-398c-15e6e83704ee}]"
2024-05-07T18:34:36.895Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:36.895Z [INFO]  server-89c13de8-7870-23a8-398c-15e6e83704ee: entering follower state: follower="Node at 89c13de8-7870-23a8-398c-15e6e83704ee [Follower]" leader-address= leader-id=
2024-05-07T18:34:36.986Z [WARN]  server-89c13de8-7870-23a8-398c-15e6e83704ee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:36.986Z [INFO]  server-89c13de8-7870-23a8-398c-15e6e83704ee: entering candidate state: node="Node at 89c13de8-7870-23a8-398c-15e6e83704ee [Candidate]" term=2
2024-05-07T18:34:36.986Z [INFO]  server-89c13de8-7870-23a8-398c-15e6e83704ee: election won: term=2 tally=1
2024-05-07T18:34:36.986Z [INFO]  server-89c13de8-7870-23a8-398c-15e6e83704ee: entering leader state: leader="Node at 89c13de8-7870-23a8-398c-15e6e83704ee [Leader]"
2024-05-07T18:34:37.094Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:36.986653519 +0000 UTC m=+14.587771759 (1 nodes), 91.178316ms from start of poll, 91.264787ms from cluster start. Timeout at 2024-05-07 18:34:37.094518173 +0000 UTC m=+14.695636413, 107.864654ms after stability
2024-05-07T18:34:37.095Z [INFO]  server-89c13de8-7870-23a8-398c-15e6e83704ee: starting snapshot up to: index=102
2024-05-07T18:34:37.095Z [INFO]  creating new snapshot: path=/tmp/raft3841720711/snapshots/2-102-1715106877095.tmp
2024-05-07T18:34:37.096Z [INFO]  server-89c13de8-7870-23a8-398c-15e6e83704ee: compacting logs: from=1 to=92
2024-05-07T18:34:37.096Z [INFO]  server-89c13de8-7870-23a8-398c-15e6e83704ee: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.21s)
=== RUN   TestRaft_NoRestoreOnStart
2024-05-07T18:34:37.102Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba Address:72d6952a-1e68-5cc6-251f-3491b8a1d4ba}]"
2024-05-07T18:34:37.102Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:37.102Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: entering follower state: follower="Node at 72d6952a-1e68-5cc6-251f-3491b8a1d4ba [Follower]" leader-address= leader-id=
2024-05-07T18:34:37.197Z [WARN]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:37.197Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: entering candidate state: node="Node at 72d6952a-1e68-5cc6-251f-3491b8a1d4ba [Candidate]" term=2
2024-05-07T18:34:37.197Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: election won: term=2 tally=1
2024-05-07T18:34:37.197Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: entering leader state: leader="Node at 72d6952a-1e68-5cc6-251f-3491b8a1d4ba [Leader]"
2024-05-07T18:34:37.306Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:37.197687833 +0000 UTC m=+14.798806073 (1 nodes), 94.779566ms from start of poll, 94.874752ms from cluster start. Timeout at 2024-05-07 18:34:37.3064908 +0000 UTC m=+14.907609040, 108.802967ms after stability
2024-05-07T18:34:37.307Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: starting snapshot up to: index=102
2024-05-07T18:34:37.307Z [INFO]  creating new snapshot: path=/tmp/raft2992224515/snapshots/2-102-1715106877307.tmp
2024-05-07T18:34:37.307Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: compacting logs: from=1 to=92
2024-05-07T18:34:37.307Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: snapshot complete up to: index=102
2024-05-07T18:34:37.307Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba Address:72d6952a-1e68-5cc6-251f-3491b8a1d4ba}]"
--- PASS: TestRaft_NoRestoreOnStart (0.21s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2024-05-07T18:34:37.308Z [INFO]  server-c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e Address:c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e}]"
2024-05-07T18:34:37.308Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:37.308Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: entering follower state: follower="Node at 72d6952a-1e68-5cc6-251f-3491b8a1d4ba [Follower]" leader-address= leader-id=
2024-05-07T18:34:37.308Z [INFO]  server-c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e: entering follower state: follower="Node at c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e [Follower]" leader-address= leader-id=
2024-05-07T18:34:37.394Z [WARN]  server-c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:37.394Z [INFO]  server-c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e: entering candidate state: node="Node at c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e [Candidate]" term=2
2024-05-07T18:34:37.394Z [WARN]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:37.394Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: entering candidate state: node="Node at 72d6952a-1e68-5cc6-251f-3491b8a1d4ba [Candidate]" term=3
2024-05-07T18:34:37.394Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: election won: term=3 tally=1
2024-05-07T18:34:37.394Z [INFO]  server-72d6952a-1e68-5cc6-251f-3491b8a1d4ba: entering leader state: leader="Node at 72d6952a-1e68-5cc6-251f-3491b8a1d4ba [Leader]"
2024-05-07T18:34:37.394Z [INFO]  server-c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e: election won: term=2 tally=1
2024-05-07T18:34:37.394Z [INFO]  server-c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e: entering leader state: leader="Node at c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e [Leader]"
2024-05-07T18:34:37.502Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:37.394704833 +0000 UTC m=+14.995823073 (1 nodes), 86.589782ms from start of poll, 86.659191ms from cluster start. Timeout at 2024-05-07 18:34:37.502489698 +0000 UTC m=+15.103607938, 107.784865ms after stability
2024-05-07T18:34:37.510Z [INFO]  server-c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e: starting snapshot up to: index=102
2024-05-07T18:34:37.510Z [INFO]  creating new snapshot: path=/tmp/raft3612724233/snapshots/2-102-1715106877510.tmp
2024-05-07T18:34:37.511Z [INFO]  server-c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e: compacting logs: from=1 to=92
2024-05-07T18:34:37.511Z [INFO]  server-c6bb15c6-9dc6-1e7a-776a-ec2a8b8ce16e: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.30s)
=== RUN   TestRaft_UserSnapshot
2024-05-07T18:34:37.611Z [INFO]  server-f2eed3ef-60f4-5e9c-def7-4d9def190284: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f2eed3ef-60f4-5e9c-def7-4d9def190284 Address:f2eed3ef-60f4-5e9c-def7-4d9def190284}]"
2024-05-07T18:34:37.611Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:37.611Z [INFO]  server-f2eed3ef-60f4-5e9c-def7-4d9def190284: entering follower state: follower="Node at f2eed3ef-60f4-5e9c-def7-4d9def190284 [Follower]" leader-address= leader-id=
2024-05-07T18:34:37.674Z [WARN]  server-f2eed3ef-60f4-5e9c-def7-4d9def190284: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:37.674Z [INFO]  server-f2eed3ef-60f4-5e9c-def7-4d9def190284: entering candidate state: node="Node at f2eed3ef-60f4-5e9c-def7-4d9def190284 [Candidate]" term=2
2024-05-07T18:34:37.674Z [INFO]  server-f2eed3ef-60f4-5e9c-def7-4d9def190284: election won: term=2 tally=1
2024-05-07T18:34:37.674Z [INFO]  server-f2eed3ef-60f4-5e9c-def7-4d9def190284: entering leader state: leader="Node at f2eed3ef-60f4-5e9c-def7-4d9def190284 [Leader]"
2024-05-07T18:34:37.782Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:37.674656819 +0000 UTC m=+15.275775059 (1 nodes), 63.481442ms from start of poll, 63.589573ms from cluster start. Timeout at 2024-05-07 18:34:37.782490504 +0000 UTC m=+15.383608744, 107.833685ms after stability
2024-05-07T18:34:37.782Z [ERROR] server-f2eed3ef-60f4-5e9c-def7-4d9def190284: failed to take snapshot: error="nothing new to snapshot"
2024-05-07T18:34:37.782Z [INFO]  server-f2eed3ef-60f4-5e9c-def7-4d9def190284: starting snapshot up to: index=12
2024-05-07T18:34:37.782Z [INFO]  creating new snapshot: path=/tmp/raft1938397171/snapshots/2-12-1715106877782.tmp
2024-05-07T18:34:37.783Z [INFO]  server-f2eed3ef-60f4-5e9c-def7-4d9def190284: compacting logs: from=1 to=2
2024-05-07T18:34:37.783Z [INFO]  server-f2eed3ef-60f4-5e9c-def7-4d9def190284: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.18s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2024-05-07T18:34:37.787Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3 Address:b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3} {Suffrage:Voter ID:server-ef79ebe3-6d2e-5000-5565-14758ca1f8df Address:ef79ebe3-6d2e-5000-5565-14758ca1f8df} {Suffrage:Voter ID:server-48b03744-b981-c07f-03ff-ec3a241d74cb Address:48b03744-b981-c07f-03ff-ec3a241d74cb}]"
2024-05-07T18:34:37.787Z [INFO]  server-ef79ebe3-6d2e-5000-5565-14758ca1f8df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3 Address:b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3} {Suffrage:Voter ID:server-ef79ebe3-6d2e-5000-5565-14758ca1f8df Address:ef79ebe3-6d2e-5000-5565-14758ca1f8df} {Suffrage:Voter ID:server-48b03744-b981-c07f-03ff-ec3a241d74cb Address:48b03744-b981-c07f-03ff-ec3a241d74cb}]"
2024-05-07T18:34:37.787Z [INFO]  server-48b03744-b981-c07f-03ff-ec3a241d74cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3 Address:b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3} {Suffrage:Voter ID:server-ef79ebe3-6d2e-5000-5565-14758ca1f8df Address:ef79ebe3-6d2e-5000-5565-14758ca1f8df} {Suffrage:Voter ID:server-48b03744-b981-c07f-03ff-ec3a241d74cb Address:48b03744-b981-c07f-03ff-ec3a241d74cb}]"
2024-05-07T18:34:37.787Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:37.787Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: entering follower state: follower="Node at b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3 [Follower]" leader-address= leader-id=
2024-05-07T18:34:37.787Z [INFO]  server-ef79ebe3-6d2e-5000-5565-14758ca1f8df: entering follower state: follower="Node at ef79ebe3-6d2e-5000-5565-14758ca1f8df [Follower]" leader-address= leader-id=
2024-05-07T18:34:37.787Z [INFO]  server-48b03744-b981-c07f-03ff-ec3a241d74cb: entering follower state: follower="Node at 48b03744-b981-c07f-03ff-ec3a241d74cb [Follower]" leader-address= leader-id=
2024-05-07T18:34:38.358Z [WARN]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:38.358Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: entering candidate state: node="Node at b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3 [Candidate]" term=2
2024-05-07T18:34:38.358Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: election won: term=2 tally=2
2024-05-07T18:34:38.358Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: entering leader state: leader="Node at b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3 [Leader]"
2024-05-07T18:34:38.358Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: added peer, starting replication: peer=server-ef79ebe3-6d2e-5000-5565-14758ca1f8df
2024-05-07T18:34:38.358Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: added peer, starting replication: peer=server-48b03744-b981-c07f-03ff-ec3a241d74cb
2024-05-07T18:34:38.358Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: pipelining replication: peer="{Voter server-48b03744-b981-c07f-03ff-ec3a241d74cb 48b03744-b981-c07f-03ff-ec3a241d74cb}"
2024-05-07T18:34:38.358Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: pipelining replication: peer="{Voter server-ef79ebe3-6d2e-5000-5565-14758ca1f8df ef79ebe3-6d2e-5000-5565-14758ca1f8df}"
2024-05-07T18:34:39.370Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:38.358814565 +0000 UTC m=+15.959932805 (1 nodes), 571.652212ms from start of poll, 571.854769ms from cluster start. Timeout at 2024-05-07 18:34:39.370492892 +0000 UTC m=+16.971611132, 1.011678327s after stability
2024-05-07T18:34:39.370Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: starting snapshot up to: index=12
2024-05-07T18:34:39.370Z [INFO]  creating new snapshot: path=/tmp/raft302154579/snapshots/2-12-1715106879370.tmp
2024-05-07T18:34:39.371Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: snapshot complete up to: index=12
2024-05-07T18:34:39.371Z [INFO]  creating new snapshot: path=/tmp/raft302154579/snapshots/2-23-1715106879371.tmp
2024-05-07T18:34:39.372Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: copied to local snapshot: bytes=71
2024-05-07T18:34:39.372Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: snapshot restore progress: id=2-23-1715106879371 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:39.372Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: restored user snapshot: index=23
2024-05-07T18:34:39.372Z [ERROR] server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: failed to get log: index=23 error="log not found"
2024-05-07T18:34:39.372Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: aborting pipeline replication: peer="{Voter server-48b03744-b981-c07f-03ff-ec3a241d74cb 48b03744-b981-c07f-03ff-ec3a241d74cb}"
2024-05-07T18:34:39.372Z [ERROR] server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: failed to get log: index=23 error="log not found"
2024-05-07T18:34:39.372Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: aborting pipeline replication: peer="{Voter server-ef79ebe3-6d2e-5000-5565-14758ca1f8df ef79ebe3-6d2e-5000-5565-14758ca1f8df}"
2024-05-07T18:34:39.378Z [ERROR] server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: failed to get log: index=23 error="log not found"
2024-05-07T18:34:39.378Z [INFO]  creating new snapshot: path=/tmp/raft3858298158/snapshots/2-23-1715106879378.tmp
2024-05-07T18:34:39.379Z [INFO]  server-ef79ebe3-6d2e-5000-5565-14758ca1f8df: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:39.379Z [INFO]  server-ef79ebe3-6d2e-5000-5565-14758ca1f8df: copied to local snapshot: bytes=71
2024-05-07T18:34:39.379Z [INFO]  server-ef79ebe3-6d2e-5000-5565-14758ca1f8df: snapshot restore progress: id=2-23-1715106879378 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:39.379Z [INFO]  server-ef79ebe3-6d2e-5000-5565-14758ca1f8df: Installed remote snapshot
2024-05-07T18:34:39.379Z [WARN]  server-ef79ebe3-6d2e-5000-5565-14758ca1f8df: skipping application of old log: index=22
2024-05-07T18:34:39.379Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: pipelining replication: peer="{Voter server-ef79ebe3-6d2e-5000-5565-14758ca1f8df ef79ebe3-6d2e-5000-5565-14758ca1f8df}"
2024-05-07T18:34:39.379Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:39.379Z [ERROR] server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: failed to get log: index=23 error="log not found"
2024-05-07T18:34:39.379Z [INFO]  creating new snapshot: path=/tmp/raft3260500899/snapshots/2-23-1715106879379.tmp
2024-05-07T18:34:39.379Z [INFO]  server-48b03744-b981-c07f-03ff-ec3a241d74cb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:39.379Z [INFO]  server-48b03744-b981-c07f-03ff-ec3a241d74cb: copied to local snapshot: bytes=71
2024-05-07T18:34:39.380Z [INFO]  server-48b03744-b981-c07f-03ff-ec3a241d74cb: snapshot restore progress: id=2-23-1715106879379 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:39.380Z [INFO]  server-48b03744-b981-c07f-03ff-ec3a241d74cb: Installed remote snapshot
2024-05-07T18:34:39.380Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: pipelining replication: peer="{Voter server-48b03744-b981-c07f-03ff-ec3a241d74cb 48b03744-b981-c07f-03ff-ec3a241d74cb}"
2024-05-07T18:34:40.386Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:39.379465491 +0000 UTC m=+16.980583731 (2 nodes), 662ns from start of poll, 1.592505695s from cluster start. Timeout at 2024-05-07 18:34:40.3864838 +0000 UTC m=+17.987602040, 1.007018309s after stability
2024-05-07T18:34:40.402Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: aborting pipeline replication: peer="{Voter server-48b03744-b981-c07f-03ff-ec3a241d74cb 48b03744-b981-c07f-03ff-ec3a241d74cb}"
2024-05-07T18:34:40.402Z [INFO]  server-b36a7a13-d35f-3a3e-2fe8-da9f3b8516b3: aborting pipeline replication: peer="{Voter server-ef79ebe3-6d2e-5000-5565-14758ca1f8df ef79ebe3-6d2e-5000-5565-14758ca1f8df}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2024-05-07T18:34:40.403Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ace295d-5eda-885c-f1ee-cc34cb615178 Address:5ace295d-5eda-885c-f1ee-cc34cb615178} {Suffrage:Voter ID:server-af650b7c-adbc-f922-4d26-689aad9614c1 Address:af650b7c-adbc-f922-4d26-689aad9614c1} {Suffrage:Voter ID:server-65392c96-0bb3-4b81-d721-8de1b103a513 Address:65392c96-0bb3-4b81-d721-8de1b103a513}]"
2024-05-07T18:34:40.403Z [INFO]  server-af650b7c-adbc-f922-4d26-689aad9614c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ace295d-5eda-885c-f1ee-cc34cb615178 Address:5ace295d-5eda-885c-f1ee-cc34cb615178} {Suffrage:Voter ID:server-af650b7c-adbc-f922-4d26-689aad9614c1 Address:af650b7c-adbc-f922-4d26-689aad9614c1} {Suffrage:Voter ID:server-65392c96-0bb3-4b81-d721-8de1b103a513 Address:65392c96-0bb3-4b81-d721-8de1b103a513}]"
2024-05-07T18:34:40.403Z [INFO]  server-65392c96-0bb3-4b81-d721-8de1b103a513: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ace295d-5eda-885c-f1ee-cc34cb615178 Address:5ace295d-5eda-885c-f1ee-cc34cb615178} {Suffrage:Voter ID:server-af650b7c-adbc-f922-4d26-689aad9614c1 Address:af650b7c-adbc-f922-4d26-689aad9614c1} {Suffrage:Voter ID:server-65392c96-0bb3-4b81-d721-8de1b103a513 Address:65392c96-0bb3-4b81-d721-8de1b103a513}]"
2024-05-07T18:34:40.403Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:40.403Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: entering follower state: follower="Node at 5ace295d-5eda-885c-f1ee-cc34cb615178 [Follower]" leader-address= leader-id=
2024-05-07T18:34:40.403Z [INFO]  server-af650b7c-adbc-f922-4d26-689aad9614c1: entering follower state: follower="Node at af650b7c-adbc-f922-4d26-689aad9614c1 [Follower]" leader-address= leader-id=
2024-05-07T18:34:40.403Z [INFO]  server-65392c96-0bb3-4b81-d721-8de1b103a513: entering follower state: follower="Node at 65392c96-0bb3-4b81-d721-8de1b103a513 [Follower]" leader-address= leader-id=
2024-05-07T18:34:41.066Z [WARN]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:41.066Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: entering candidate state: node="Node at 5ace295d-5eda-885c-f1ee-cc34cb615178 [Candidate]" term=2
2024-05-07T18:34:41.066Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: election won: term=2 tally=2
2024-05-07T18:34:41.066Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: entering leader state: leader="Node at 5ace295d-5eda-885c-f1ee-cc34cb615178 [Leader]"
2024-05-07T18:34:41.066Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: added peer, starting replication: peer=server-af650b7c-adbc-f922-4d26-689aad9614c1
2024-05-07T18:34:41.066Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: added peer, starting replication: peer=server-65392c96-0bb3-4b81-d721-8de1b103a513
2024-05-07T18:34:41.066Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: pipelining replication: peer="{Voter server-65392c96-0bb3-4b81-d721-8de1b103a513 65392c96-0bb3-4b81-d721-8de1b103a513}"
2024-05-07T18:34:41.066Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: pipelining replication: peer="{Voter server-af650b7c-adbc-f922-4d26-689aad9614c1 af650b7c-adbc-f922-4d26-689aad9614c1}"
2024-05-07T18:34:42.074Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:41.066801524 +0000 UTC m=+18.667919774 (1 nodes), 663.125988ms from start of poll, 663.304461ms from cluster start. Timeout at 2024-05-07 18:34:42.074585017 +0000 UTC m=+19.675703257, 1.007783483s after stability
2024-05-07T18:34:42.074Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: starting snapshot up to: index=12
2024-05-07T18:34:42.074Z [INFO]  creating new snapshot: path=/tmp/raft3983102781/snapshots/2-12-1715106882074.tmp
2024-05-07T18:34:42.075Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: snapshot complete up to: index=12
2024-05-07T18:34:42.075Z [INFO]  creating new snapshot: path=/tmp/raft3983102781/snapshots/2-23-1715106882075.tmp
2024-05-07T18:34:42.076Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: copied to local snapshot: bytes=71
2024-05-07T18:34:42.076Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: snapshot restore progress: id=2-23-1715106882075 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:42.076Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: removing all old logs from log store
2024-05-07T18:34:42.076Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: compacting logs: from=1 to=22
2024-05-07T18:34:42.076Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: restored user snapshot: index=23
2024-05-07T18:34:42.076Z [ERROR] server-5ace295d-5eda-885c-f1ee-cc34cb615178: failed to get log: index=22 error="log not found"
2024-05-07T18:34:42.076Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: aborting pipeline replication: peer="{Voter server-65392c96-0bb3-4b81-d721-8de1b103a513 65392c96-0bb3-4b81-d721-8de1b103a513}"
2024-05-07T18:34:42.076Z [ERROR] server-5ace295d-5eda-885c-f1ee-cc34cb615178: failed to get log: index=22 error="log not found"
2024-05-07T18:34:42.076Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: aborting pipeline replication: peer="{Voter server-af650b7c-adbc-f922-4d26-689aad9614c1 af650b7c-adbc-f922-4d26-689aad9614c1}"
2024-05-07T18:34:42.082Z [ERROR] server-5ace295d-5eda-885c-f1ee-cc34cb615178: failed to get log: index=22 error="log not found"
2024-05-07T18:34:42.082Z [INFO]  creating new snapshot: path=/tmp/raft1821524024/snapshots/2-23-1715106882082.tmp
2024-05-07T18:34:42.083Z [INFO]  server-65392c96-0bb3-4b81-d721-8de1b103a513: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:42.083Z [INFO]  server-65392c96-0bb3-4b81-d721-8de1b103a513: copied to local snapshot: bytes=71
2024-05-07T18:34:42.090Z [ERROR] server-5ace295d-5eda-885c-f1ee-cc34cb615178: failed to get log: index=22 error="log not found"
2024-05-07T18:34:42.090Z [INFO]  creating new snapshot: path=/tmp/raft3417293934/snapshots/2-23-1715106882090.tmp
2024-05-07T18:34:42.090Z [INFO]  server-af650b7c-adbc-f922-4d26-689aad9614c1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:42.091Z [INFO]  server-af650b7c-adbc-f922-4d26-689aad9614c1: copied to local snapshot: bytes=71
2024-05-07T18:34:42.091Z [INFO]  server-af650b7c-adbc-f922-4d26-689aad9614c1: snapshot restore progress: id=2-23-1715106882090 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:42.091Z [INFO]  server-af650b7c-adbc-f922-4d26-689aad9614c1: removing all old logs from log store
2024-05-07T18:34:42.091Z [INFO]  server-af650b7c-adbc-f922-4d26-689aad9614c1: compacting logs: from=1 to=22
2024-05-07T18:34:42.091Z [INFO]  server-af650b7c-adbc-f922-4d26-689aad9614c1: Installed remote snapshot
2024-05-07T18:34:42.091Z [WARN]  server-af650b7c-adbc-f922-4d26-689aad9614c1: skipping application of old log: index=22
2024-05-07T18:34:42.091Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: pipelining replication: peer="{Voter server-af650b7c-adbc-f922-4d26-689aad9614c1 af650b7c-adbc-f922-4d26-689aad9614c1}"
2024-05-07T18:34:42.091Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:42.091Z [INFO]  server-65392c96-0bb3-4b81-d721-8de1b103a513: snapshot restore progress: id=2-23-1715106882082 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:42.091Z [INFO]  server-65392c96-0bb3-4b81-d721-8de1b103a513: removing all old logs from log store
2024-05-07T18:34:42.091Z [INFO]  server-65392c96-0bb3-4b81-d721-8de1b103a513: compacting logs: from=1 to=22
2024-05-07T18:34:42.091Z [INFO]  server-65392c96-0bb3-4b81-d721-8de1b103a513: Installed remote snapshot
2024-05-07T18:34:42.091Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: pipelining replication: peer="{Voter server-65392c96-0bb3-4b81-d721-8de1b103a513 65392c96-0bb3-4b81-d721-8de1b103a513}"
2024-05-07T18:34:43.098Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:42.091381276 +0000 UTC m=+19.692499516 (2 nodes), 771ns from start of poll, 1.687884203s from cluster start. Timeout at 2024-05-07 18:34:43.098480537 +0000 UTC m=+20.699598777, 1.007099261s after stability
2024-05-07T18:34:43.114Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: aborting pipeline replication: peer="{Voter server-65392c96-0bb3-4b81-d721-8de1b103a513 65392c96-0bb3-4b81-d721-8de1b103a513}"
2024-05-07T18:34:43.114Z [INFO]  server-5ace295d-5eda-885c-f1ee-cc34cb615178: aborting pipeline replication: peer="{Voter server-af650b7c-adbc-f922-4d26-689aad9614c1 af650b7c-adbc-f922-4d26-689aad9614c1}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2024-05-07T18:34:43.115Z [INFO]  server-d5c730a8-2bc7-a974-6765-8c9d3f985088: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5c730a8-2bc7-a974-6765-8c9d3f985088 Address:d5c730a8-2bc7-a974-6765-8c9d3f985088} {Suffrage:Voter ID:server-fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc Address:fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc} {Suffrage:Voter ID:server-53445995-b46f-ae7f-ac1d-29dec7a9193a Address:53445995-b46f-ae7f-ac1d-29dec7a9193a}]"
2024-05-07T18:34:43.115Z [INFO]  server-fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5c730a8-2bc7-a974-6765-8c9d3f985088 Address:d5c730a8-2bc7-a974-6765-8c9d3f985088} {Suffrage:Voter ID:server-fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc Address:fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc} {Suffrage:Voter ID:server-53445995-b46f-ae7f-ac1d-29dec7a9193a Address:53445995-b46f-ae7f-ac1d-29dec7a9193a}]"
2024-05-07T18:34:43.115Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5c730a8-2bc7-a974-6765-8c9d3f985088 Address:d5c730a8-2bc7-a974-6765-8c9d3f985088} {Suffrage:Voter ID:server-fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc Address:fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc} {Suffrage:Voter ID:server-53445995-b46f-ae7f-ac1d-29dec7a9193a Address:53445995-b46f-ae7f-ac1d-29dec7a9193a}]"
2024-05-07T18:34:43.115Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:43.115Z [INFO]  server-d5c730a8-2bc7-a974-6765-8c9d3f985088: entering follower state: follower="Node at d5c730a8-2bc7-a974-6765-8c9d3f985088 [Follower]" leader-address= leader-id=
2024-05-07T18:34:43.115Z [INFO]  server-fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc: entering follower state: follower="Node at fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc [Follower]" leader-address= leader-id=
2024-05-07T18:34:43.115Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: entering follower state: follower="Node at 53445995-b46f-ae7f-ac1d-29dec7a9193a [Follower]" leader-address= leader-id=
2024-05-07T18:34:43.714Z [WARN]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:43.714Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: entering candidate state: node="Node at 53445995-b46f-ae7f-ac1d-29dec7a9193a [Candidate]" term=2
2024-05-07T18:34:43.714Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: election won: term=2 tally=2
2024-05-07T18:34:43.714Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: entering leader state: leader="Node at 53445995-b46f-ae7f-ac1d-29dec7a9193a [Leader]"
2024-05-07T18:34:43.714Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: added peer, starting replication: peer=server-d5c730a8-2bc7-a974-6765-8c9d3f985088
2024-05-07T18:34:43.714Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: added peer, starting replication: peer=server-fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc
2024-05-07T18:34:43.714Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: pipelining replication: peer="{Voter server-fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc}"
2024-05-07T18:34:43.714Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: pipelining replication: peer="{Voter server-d5c730a8-2bc7-a974-6765-8c9d3f985088 d5c730a8-2bc7-a974-6765-8c9d3f985088}"
2024-05-07T18:34:44.722Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:43.714858486 +0000 UTC m=+21.315976727 (1 nodes), 599.168581ms from start of poll, 599.342986ms from cluster start. Timeout at 2024-05-07 18:34:44.722499153 +0000 UTC m=+22.323617403, 1.007640676s after stability
2024-05-07T18:34:44.723Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: starting snapshot up to: index=12
2024-05-07T18:34:44.723Z [INFO]  creating new snapshot: path=/tmp/raft2674668907/snapshots/2-12-1715106884723.tmp
2024-05-07T18:34:44.786Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: snapshot complete up to: index=12
2024-05-07T18:34:44.787Z [INFO]  server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321 Address:76f3fab6-bbbe-bb15-c965-8c1c6ef73321} {Suffrage:Voter ID:server-ab759915-bc42-fc29-179d-126a33832ca5 Address:ab759915-bc42-fc29-179d-126a33832ca5} {Suffrage:Voter ID:server-13c6734b-1795-f1c6-c6de-c78e49e41c74 Address:13c6734b-1795-f1c6-c6de-c78e49e41c74}]"
2024-05-07T18:34:44.788Z [INFO]  server-ab759915-bc42-fc29-179d-126a33832ca5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321 Address:76f3fab6-bbbe-bb15-c965-8c1c6ef73321} {Suffrage:Voter ID:server-ab759915-bc42-fc29-179d-126a33832ca5 Address:ab759915-bc42-fc29-179d-126a33832ca5} {Suffrage:Voter ID:server-13c6734b-1795-f1c6-c6de-c78e49e41c74 Address:13c6734b-1795-f1c6-c6de-c78e49e41c74}]"
2024-05-07T18:34:44.788Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321 Address:76f3fab6-bbbe-bb15-c965-8c1c6ef73321} {Suffrage:Voter ID:server-ab759915-bc42-fc29-179d-126a33832ca5 Address:ab759915-bc42-fc29-179d-126a33832ca5} {Suffrage:Voter ID:server-13c6734b-1795-f1c6-c6de-c78e49e41c74 Address:13c6734b-1795-f1c6-c6de-c78e49e41c74}]"
2024-05-07T18:34:44.788Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:44.788Z [INFO]  server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321: entering follower state: follower="Node at 76f3fab6-bbbe-bb15-c965-8c1c6ef73321 [Follower]" leader-address= leader-id=
2024-05-07T18:34:44.788Z [INFO]  server-ab759915-bc42-fc29-179d-126a33832ca5: entering follower state: follower="Node at ab759915-bc42-fc29-179d-126a33832ca5 [Follower]" leader-address= leader-id=
2024-05-07T18:34:44.788Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: entering follower state: follower="Node at 13c6734b-1795-f1c6-c6de-c78e49e41c74 [Follower]" leader-address= leader-id=
2024-05-07T18:34:45.382Z [WARN]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:45.382Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: entering candidate state: node="Node at 13c6734b-1795-f1c6-c6de-c78e49e41c74 [Candidate]" term=2
2024-05-07T18:34:45.382Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: election won: term=2 tally=2
2024-05-07T18:34:45.382Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: entering leader state: leader="Node at 13c6734b-1795-f1c6-c6de-c78e49e41c74 [Leader]"
2024-05-07T18:34:45.382Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: added peer, starting replication: peer=server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321
2024-05-07T18:34:45.382Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: added peer, starting replication: peer=server-ab759915-bc42-fc29-179d-126a33832ca5
2024-05-07T18:34:45.382Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: pipelining replication: peer="{Voter server-ab759915-bc42-fc29-179d-126a33832ca5 ab759915-bc42-fc29-179d-126a33832ca5}"
2024-05-07T18:34:45.382Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: pipelining replication: peer="{Voter server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321 76f3fab6-bbbe-bb15-c965-8c1c6ef73321}"
2024-05-07T18:34:46.394Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:45.382936872 +0000 UTC m=+22.984055122 (1 nodes), 594.831675ms from start of poll, 595.124951ms from cluster start. Timeout at 2024-05-07 18:34:46.394673176 +0000 UTC m=+23.995791416, 1.011736294s after stability
2024-05-07T18:34:46.394Z [INFO]  creating new snapshot: path=/tmp/raft2666439048/snapshots/2-13-1715106886394.tmp
2024-05-07T18:34:46.395Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: copied to local snapshot: bytes=71
2024-05-07T18:34:46.395Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: snapshot restore progress: id=2-13-1715106886394 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:46.395Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: restored user snapshot: index=13
2024-05-07T18:34:46.395Z [ERROR] server-13c6734b-1795-f1c6-c6de-c78e49e41c74: failed to get log: index=3 error="log not found"
2024-05-07T18:34:46.395Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: aborting pipeline replication: peer="{Voter server-ab759915-bc42-fc29-179d-126a33832ca5 ab759915-bc42-fc29-179d-126a33832ca5}"
2024-05-07T18:34:46.395Z [ERROR] server-13c6734b-1795-f1c6-c6de-c78e49e41c74: failed to get log: index=3 error="log not found"
2024-05-07T18:34:46.395Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: aborting pipeline replication: peer="{Voter server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321 76f3fab6-bbbe-bb15-c965-8c1c6ef73321}"
2024-05-07T18:34:46.406Z [ERROR] server-13c6734b-1795-f1c6-c6de-c78e49e41c74: failed to get log: index=3 error="log not found"
2024-05-07T18:34:46.406Z [INFO]  creating new snapshot: path=/tmp/raft1517932499/snapshots/2-13-1715106886406.tmp
2024-05-07T18:34:46.406Z [INFO]  server-ab759915-bc42-fc29-179d-126a33832ca5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:46.407Z [INFO]  server-ab759915-bc42-fc29-179d-126a33832ca5: copied to local snapshot: bytes=71
2024-05-07T18:34:46.407Z [INFO]  server-ab759915-bc42-fc29-179d-126a33832ca5: snapshot restore progress: id=2-13-1715106886406 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:46.407Z [INFO]  server-ab759915-bc42-fc29-179d-126a33832ca5: Installed remote snapshot
2024-05-07T18:34:46.407Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: pipelining replication: peer="{Voter server-ab759915-bc42-fc29-179d-126a33832ca5 ab759915-bc42-fc29-179d-126a33832ca5}"
2024-05-07T18:34:46.407Z [ERROR] server-13c6734b-1795-f1c6-c6de-c78e49e41c74: failed to get log: index=3 error="log not found"
2024-05-07T18:34:46.407Z [INFO]  creating new snapshot: path=/tmp/raft748365623/snapshots/2-13-1715106886407.tmp
2024-05-07T18:34:46.407Z [INFO]  server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:46.407Z [INFO]  server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321: copied to local snapshot: bytes=71
2024-05-07T18:34:46.407Z [INFO]  server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321: snapshot restore progress: id=2-13-1715106886407 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:46.407Z [INFO]  server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321: Installed remote snapshot
2024-05-07T18:34:46.407Z [INFO]  server-13c6734b-1795-f1c6-c6de-c78e49e41c74: pipelining replication: peer="{Voter server-76f3fab6-bbbe-bb15-c965-8c1c6ef73321 76f3fab6-bbbe-bb15-c965-8c1c6ef73321}"
2024-05-07T18:34:46.407Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:47.414Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:46.407889909 +0000 UTC m=+24.009008149 (2 nodes), 762ns from start of poll, 1.620077978s from cluster start. Timeout at 2024-05-07 18:34:47.414620122 +0000 UTC m=+25.015738372, 1.006730223s after stability
2024-05-07T18:34:47.450Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: aborting pipeline replication: peer="{Voter server-fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc fe0b23c0-f594-eb2f-dc24-cd7d0173d6dc}"
2024-05-07T18:34:47.450Z [INFO]  server-53445995-b46f-ae7f-ac1d-29dec7a9193a: aborting pipeline replication: peer="{Voter server-d5c730a8-2bc7-a974-6765-8c9d3f985088 d5c730a8-2bc7-a974-6765-8c9d3f985088}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2024-05-07T18:34:47.451Z [INFO]  server-7b298abc-f297-19a5-35d7-993442ff52c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b298abc-f297-19a5-35d7-993442ff52c5 Address:7b298abc-f297-19a5-35d7-993442ff52c5} {Suffrage:Voter ID:server-0cb5474b-0201-923e-64b5-f559973ba358 Address:0cb5474b-0201-923e-64b5-f559973ba358} {Suffrage:Voter ID:server-bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d Address:bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d}]"
2024-05-07T18:34:47.451Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b298abc-f297-19a5-35d7-993442ff52c5 Address:7b298abc-f297-19a5-35d7-993442ff52c5} {Suffrage:Voter ID:server-0cb5474b-0201-923e-64b5-f559973ba358 Address:0cb5474b-0201-923e-64b5-f559973ba358} {Suffrage:Voter ID:server-bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d Address:bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d}]"
2024-05-07T18:34:47.451Z [INFO]  server-bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b298abc-f297-19a5-35d7-993442ff52c5 Address:7b298abc-f297-19a5-35d7-993442ff52c5} {Suffrage:Voter ID:server-0cb5474b-0201-923e-64b5-f559973ba358 Address:0cb5474b-0201-923e-64b5-f559973ba358} {Suffrage:Voter ID:server-bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d Address:bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d}]"
2024-05-07T18:34:47.451Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:47.451Z [INFO]  server-7b298abc-f297-19a5-35d7-993442ff52c5: entering follower state: follower="Node at 7b298abc-f297-19a5-35d7-993442ff52c5 [Follower]" leader-address= leader-id=
2024-05-07T18:34:47.452Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: entering follower state: follower="Node at 0cb5474b-0201-923e-64b5-f559973ba358 [Follower]" leader-address= leader-id=
2024-05-07T18:34:47.452Z [INFO]  server-bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d: entering follower state: follower="Node at bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d [Follower]" leader-address= leader-id=
2024-05-07T18:34:48.052Z [WARN]  server-0cb5474b-0201-923e-64b5-f559973ba358: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:48.052Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: entering candidate state: node="Node at 0cb5474b-0201-923e-64b5-f559973ba358 [Candidate]" term=2
2024-05-07T18:34:48.052Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: election won: term=2 tally=2
2024-05-07T18:34:48.052Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: entering leader state: leader="Node at 0cb5474b-0201-923e-64b5-f559973ba358 [Leader]"
2024-05-07T18:34:48.052Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: added peer, starting replication: peer=server-7b298abc-f297-19a5-35d7-993442ff52c5
2024-05-07T18:34:48.052Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: added peer, starting replication: peer=server-bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d
2024-05-07T18:34:48.052Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: pipelining replication: peer="{Voter server-bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d}"
2024-05-07T18:34:48.052Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: pipelining replication: peer="{Voter server-7b298abc-f297-19a5-35d7-993442ff52c5 7b298abc-f297-19a5-35d7-993442ff52c5}"
2024-05-07T18:34:49.058Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:48.053003781 +0000 UTC m=+25.654122021 (1 nodes), 601.052114ms from start of poll, 601.294184ms from cluster start. Timeout at 2024-05-07 18:34:49.058388402 +0000 UTC m=+26.659506642, 1.005384621s after stability
2024-05-07T18:34:49.059Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: starting snapshot up to: index=12
2024-05-07T18:34:49.059Z [INFO]  creating new snapshot: path=/tmp/raft3664372107/snapshots/2-12-1715106889059.tmp
2024-05-07T18:34:49.067Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: snapshot complete up to: index=12
2024-05-07T18:34:49.068Z [INFO]  server-f20dd97d-8b10-db87-2a48-9d758bc1e477: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f20dd97d-8b10-db87-2a48-9d758bc1e477 Address:f20dd97d-8b10-db87-2a48-9d758bc1e477} {Suffrage:Voter ID:server-7bee73fe-23b1-babb-3312-9e8518fe7cd6 Address:7bee73fe-23b1-babb-3312-9e8518fe7cd6} {Suffrage:Voter ID:server-752999e5-af08-060e-1890-7b4150c24e29 Address:752999e5-af08-060e-1890-7b4150c24e29}]"
2024-05-07T18:34:49.068Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f20dd97d-8b10-db87-2a48-9d758bc1e477 Address:f20dd97d-8b10-db87-2a48-9d758bc1e477} {Suffrage:Voter ID:server-7bee73fe-23b1-babb-3312-9e8518fe7cd6 Address:7bee73fe-23b1-babb-3312-9e8518fe7cd6} {Suffrage:Voter ID:server-752999e5-af08-060e-1890-7b4150c24e29 Address:752999e5-af08-060e-1890-7b4150c24e29}]"
2024-05-07T18:34:49.068Z [INFO]  server-752999e5-af08-060e-1890-7b4150c24e29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f20dd97d-8b10-db87-2a48-9d758bc1e477 Address:f20dd97d-8b10-db87-2a48-9d758bc1e477} {Suffrage:Voter ID:server-7bee73fe-23b1-babb-3312-9e8518fe7cd6 Address:7bee73fe-23b1-babb-3312-9e8518fe7cd6} {Suffrage:Voter ID:server-752999e5-af08-060e-1890-7b4150c24e29 Address:752999e5-af08-060e-1890-7b4150c24e29}]"
2024-05-07T18:34:49.068Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:49.068Z [INFO]  server-f20dd97d-8b10-db87-2a48-9d758bc1e477: entering follower state: follower="Node at f20dd97d-8b10-db87-2a48-9d758bc1e477 [Follower]" leader-address= leader-id=
2024-05-07T18:34:49.068Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: entering follower state: follower="Node at 7bee73fe-23b1-babb-3312-9e8518fe7cd6 [Follower]" leader-address= leader-id=
2024-05-07T18:34:49.068Z [INFO]  server-752999e5-af08-060e-1890-7b4150c24e29: entering follower state: follower="Node at 752999e5-af08-060e-1890-7b4150c24e29 [Follower]" leader-address= leader-id=
2024-05-07T18:34:49.766Z [WARN]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:49.766Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: entering candidate state: node="Node at 7bee73fe-23b1-babb-3312-9e8518fe7cd6 [Candidate]" term=2
2024-05-07T18:34:49.766Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: election won: term=2 tally=2
2024-05-07T18:34:49.766Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: entering leader state: leader="Node at 7bee73fe-23b1-babb-3312-9e8518fe7cd6 [Leader]"
2024-05-07T18:34:49.766Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: added peer, starting replication: peer=server-f20dd97d-8b10-db87-2a48-9d758bc1e477
2024-05-07T18:34:49.766Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: added peer, starting replication: peer=server-752999e5-af08-060e-1890-7b4150c24e29
2024-05-07T18:34:49.766Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: pipelining replication: peer="{Voter server-752999e5-af08-060e-1890-7b4150c24e29 752999e5-af08-060e-1890-7b4150c24e29}"
2024-05-07T18:34:49.766Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: pipelining replication: peer="{Voter server-f20dd97d-8b10-db87-2a48-9d758bc1e477 f20dd97d-8b10-db87-2a48-9d758bc1e477}"
2024-05-07T18:34:50.774Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:49.76681121 +0000 UTC m=+27.367929450 (1 nodes), 698.620266ms from start of poll, 698.795743ms from cluster start. Timeout at 2024-05-07 18:34:50.774549158 +0000 UTC m=+28.375667398, 1.007737948s after stability
2024-05-07T18:34:50.774Z [INFO]  creating new snapshot: path=/tmp/raft1372515362/snapshots/2-13-1715106890774.tmp
2024-05-07T18:34:50.822Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: copied to local snapshot: bytes=71
2024-05-07T18:34:50.822Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: snapshot restore progress: id=2-13-1715106890774 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:50.822Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: removing all old logs from log store
2024-05-07T18:34:50.822Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: compacting logs: from=1 to=2
2024-05-07T18:34:50.822Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: restored user snapshot: index=13
2024-05-07T18:34:50.823Z [ERROR] server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: failed to get log: index=2 error="log not found"
2024-05-07T18:34:50.823Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: aborting pipeline replication: peer="{Voter server-752999e5-af08-060e-1890-7b4150c24e29 752999e5-af08-060e-1890-7b4150c24e29}"
2024-05-07T18:34:50.823Z [ERROR] server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: failed to get log: index=2 error="log not found"
2024-05-07T18:34:50.823Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: aborting pipeline replication: peer="{Voter server-f20dd97d-8b10-db87-2a48-9d758bc1e477 f20dd97d-8b10-db87-2a48-9d758bc1e477}"
2024-05-07T18:34:50.828Z [ERROR] server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: failed to get log: index=2 error="log not found"
2024-05-07T18:34:50.831Z [ERROR] server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: failed to get log: index=2 error="log not found"
2024-05-07T18:34:50.832Z [INFO]  creating new snapshot: path=/tmp/raft1673604669/snapshots/2-13-1715106890832.tmp
2024-05-07T18:34:50.902Z [INFO]  server-752999e5-af08-060e-1890-7b4150c24e29: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:50.903Z [INFO]  server-752999e5-af08-060e-1890-7b4150c24e29: copied to local snapshot: bytes=71
2024-05-07T18:34:50.903Z [INFO]  server-752999e5-af08-060e-1890-7b4150c24e29: snapshot restore progress: id=2-13-1715106890832 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:50.903Z [INFO]  server-752999e5-af08-060e-1890-7b4150c24e29: removing all old logs from log store
2024-05-07T18:34:50.903Z [INFO]  server-752999e5-af08-060e-1890-7b4150c24e29: compacting logs: from=1 to=2
2024-05-07T18:34:50.903Z [INFO]  server-752999e5-af08-060e-1890-7b4150c24e29: Installed remote snapshot
2024-05-07T18:34:50.903Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: pipelining replication: peer="{Voter server-752999e5-af08-060e-1890-7b4150c24e29 752999e5-af08-060e-1890-7b4150c24e29}"
2024-05-07T18:34:50.903Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:50.903Z [INFO]  creating new snapshot: path=/tmp/raft4103277361/snapshots/2-13-1715106890903.tmp
2024-05-07T18:34:50.910Z [INFO]  server-f20dd97d-8b10-db87-2a48-9d758bc1e477: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:51.027Z [INFO]  server-f20dd97d-8b10-db87-2a48-9d758bc1e477: copied to local snapshot: bytes=71
2024-05-07T18:34:51.027Z [INFO]  server-f20dd97d-8b10-db87-2a48-9d758bc1e477: snapshot restore progress: id=2-13-1715106890903 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:51.027Z [INFO]  server-f20dd97d-8b10-db87-2a48-9d758bc1e477: removing all old logs from log store
2024-05-07T18:34:51.027Z [INFO]  server-f20dd97d-8b10-db87-2a48-9d758bc1e477: compacting logs: from=1 to=2
2024-05-07T18:34:51.027Z [INFO]  server-f20dd97d-8b10-db87-2a48-9d758bc1e477: Installed remote snapshot
2024-05-07T18:34:51.027Z [INFO]  server-7bee73fe-23b1-babb-3312-9e8518fe7cd6: pipelining replication: peer="{Voter server-f20dd97d-8b10-db87-2a48-9d758bc1e477 f20dd97d-8b10-db87-2a48-9d758bc1e477}"
2024-05-07T18:34:51.910Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:50.903466778 +0000 UTC m=+28.504585028 (2 nodes), 1.132µs from start of poll, 1.835451321s from cluster start. Timeout at 2024-05-07 18:34:51.910488332 +0000 UTC m=+29.511606582, 1.007021554s after stability
2024-05-07T18:34:51.926Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: aborting pipeline replication: peer="{Voter server-bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d bdd9fcb9-7def-e97b-93fd-c0b7fa615b1d}"
2024-05-07T18:34:51.926Z [INFO]  server-0cb5474b-0201-923e-64b5-f559973ba358: aborting pipeline replication: peer="{Voter server-7b298abc-f297-19a5-35d7-993442ff52c5 7b298abc-f297-19a5-35d7-993442ff52c5}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2024-05-07T18:34:51.927Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b Address:2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b} {Suffrage:Voter ID:server-752267a0-eb28-22c6-c2fb-96f5a05734a0 Address:752267a0-eb28-22c6-c2fb-96f5a05734a0} {Suffrage:Voter ID:server-c86b4be9-20b8-d57e-9703-af94b5814ac7 Address:c86b4be9-20b8-d57e-9703-af94b5814ac7}]"
2024-05-07T18:34:51.927Z [INFO]  server-752267a0-eb28-22c6-c2fb-96f5a05734a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b Address:2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b} {Suffrage:Voter ID:server-752267a0-eb28-22c6-c2fb-96f5a05734a0 Address:752267a0-eb28-22c6-c2fb-96f5a05734a0} {Suffrage:Voter ID:server-c86b4be9-20b8-d57e-9703-af94b5814ac7 Address:c86b4be9-20b8-d57e-9703-af94b5814ac7}]"
2024-05-07T18:34:51.927Z [INFO]  server-c86b4be9-20b8-d57e-9703-af94b5814ac7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b Address:2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b} {Suffrage:Voter ID:server-752267a0-eb28-22c6-c2fb-96f5a05734a0 Address:752267a0-eb28-22c6-c2fb-96f5a05734a0} {Suffrage:Voter ID:server-c86b4be9-20b8-d57e-9703-af94b5814ac7 Address:c86b4be9-20b8-d57e-9703-af94b5814ac7}]"
2024-05-07T18:34:51.927Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:51.927Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: entering follower state: follower="Node at 2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b [Follower]" leader-address= leader-id=
2024-05-07T18:34:51.928Z [INFO]  server-752267a0-eb28-22c6-c2fb-96f5a05734a0: entering follower state: follower="Node at 752267a0-eb28-22c6-c2fb-96f5a05734a0 [Follower]" leader-address= leader-id=
2024-05-07T18:34:51.928Z [INFO]  server-c86b4be9-20b8-d57e-9703-af94b5814ac7: entering follower state: follower="Node at c86b4be9-20b8-d57e-9703-af94b5814ac7 [Follower]" leader-address= leader-id=
2024-05-07T18:34:52.602Z [WARN]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:52.602Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: entering candidate state: node="Node at 2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b [Candidate]" term=2
2024-05-07T18:34:52.602Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: election won: term=2 tally=2
2024-05-07T18:34:52.602Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: entering leader state: leader="Node at 2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b [Leader]"
2024-05-07T18:34:52.602Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: added peer, starting replication: peer=server-752267a0-eb28-22c6-c2fb-96f5a05734a0
2024-05-07T18:34:52.602Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: added peer, starting replication: peer=server-c86b4be9-20b8-d57e-9703-af94b5814ac7
2024-05-07T18:34:52.602Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: pipelining replication: peer="{Voter server-c86b4be9-20b8-d57e-9703-af94b5814ac7 c86b4be9-20b8-d57e-9703-af94b5814ac7}"
2024-05-07T18:34:52.602Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: pipelining replication: peer="{Voter server-752267a0-eb28-22c6-c2fb-96f5a05734a0 752267a0-eb28-22c6-c2fb-96f5a05734a0}"
2024-05-07T18:34:53.625Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:52.603013321 +0000 UTC m=+30.204131571 (1 nodes), 675.058582ms from start of poll, 675.306212ms from cluster start. Timeout at 2024-05-07 18:34:53.625855102 +0000 UTC m=+31.226973352, 1.022841781s after stability
2024-05-07T18:34:53.626Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: starting snapshot up to: index=12
2024-05-07T18:34:53.626Z [INFO]  creating new snapshot: path=/tmp/raft2632380371/snapshots/2-12-1715106893626.tmp
2024-05-07T18:34:53.841Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: snapshot complete up to: index=12
2024-05-07T18:34:53.842Z [INFO]  creating new snapshot: path=/tmp/raft2632380371/snapshots/2-23-1715106893842.tmp
2024-05-07T18:34:54.026Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: copied to local snapshot: bytes=71
2024-05-07T18:34:54.027Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: snapshot restore progress: id=2-23-1715106893842 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:54.027Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: restored user snapshot: index=23
2024-05-07T18:34:54.027Z [ERROR] server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: failed to get log: index=23 error="log not found"
2024-05-07T18:34:54.027Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: aborting pipeline replication: peer="{Voter server-c86b4be9-20b8-d57e-9703-af94b5814ac7 c86b4be9-20b8-d57e-9703-af94b5814ac7}"
2024-05-07T18:34:54.027Z [ERROR] server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: failed to get log: index=23 error="log not found"
2024-05-07T18:34:54.027Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: aborting pipeline replication: peer="{Voter server-752267a0-eb28-22c6-c2fb-96f5a05734a0 752267a0-eb28-22c6-c2fb-96f5a05734a0}"
2024-05-07T18:34:54.027Z [ERROR] server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: failed to get log: index=23 error="log not found"
2024-05-07T18:34:54.027Z [INFO]  creating new snapshot: path=/tmp/raft64246416/snapshots/2-23-1715106894027.tmp
2024-05-07T18:34:54.027Z [INFO]  server-752267a0-eb28-22c6-c2fb-96f5a05734a0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:54.038Z [ERROR] server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: failed to get log: index=23 error="log not found"
2024-05-07T18:34:54.038Z [INFO]  creating new snapshot: path=/tmp/raft2303388316/snapshots/2-23-1715106894038.tmp
2024-05-07T18:34:54.039Z [INFO]  server-c86b4be9-20b8-d57e-9703-af94b5814ac7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:54.076Z [INFO]  server-752267a0-eb28-22c6-c2fb-96f5a05734a0: copied to local snapshot: bytes=71
2024-05-07T18:34:54.077Z [INFO]  server-752267a0-eb28-22c6-c2fb-96f5a05734a0: snapshot restore progress: id=2-23-1715106894027 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:54.077Z [INFO]  server-752267a0-eb28-22c6-c2fb-96f5a05734a0: Installed remote snapshot
2024-05-07T18:34:54.077Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: pipelining replication: peer="{Voter server-752267a0-eb28-22c6-c2fb-96f5a05734a0 752267a0-eb28-22c6-c2fb-96f5a05734a0}"
2024-05-07T18:34:54.077Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:54.077Z [INFO]  server-c86b4be9-20b8-d57e-9703-af94b5814ac7: copied to local snapshot: bytes=71
2024-05-07T18:34:54.077Z [INFO]  server-c86b4be9-20b8-d57e-9703-af94b5814ac7: snapshot restore progress: id=2-23-1715106894038 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:54.077Z [INFO]  server-c86b4be9-20b8-d57e-9703-af94b5814ac7: Installed remote snapshot
2024-05-07T18:34:54.077Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: pipelining replication: peer="{Voter server-c86b4be9-20b8-d57e-9703-af94b5814ac7 c86b4be9-20b8-d57e-9703-af94b5814ac7}"
2024-05-07T18:34:55.083Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:54.077256453 +0000 UTC m=+31.678374703 (2 nodes), 872ns from start of poll, 2.149549344s from cluster start. Timeout at 2024-05-07 18:34:55.083653227 +0000 UTC m=+32.684771477, 1.006396774s after stability
2024-05-07T18:34:55.094Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: aborting pipeline replication: peer="{Voter server-c86b4be9-20b8-d57e-9703-af94b5814ac7 c86b4be9-20b8-d57e-9703-af94b5814ac7}"
2024-05-07T18:34:55.094Z [INFO]  server-2b2b7f85-5b47-0dc4-073d-6324a1ee6a2b: aborting pipeline replication: peer="{Voter server-752267a0-eb28-22c6-c2fb-96f5a05734a0 752267a0-eb28-22c6-c2fb-96f5a05734a0}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2024-05-07T18:34:55.095Z [INFO]  server-cff199b2-1ad0-64f2-0a0c-488f90c30f72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cff199b2-1ad0-64f2-0a0c-488f90c30f72 Address:cff199b2-1ad0-64f2-0a0c-488f90c30f72} {Suffrage:Voter ID:server-5afe6099-022d-d30b-6dd8-a6cdda2262f0 Address:5afe6099-022d-d30b-6dd8-a6cdda2262f0} {Suffrage:Voter ID:server-768b639f-e5de-e062-fbcf-7f097b2c5398 Address:768b639f-e5de-e062-fbcf-7f097b2c5398}]"
2024-05-07T18:34:55.095Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cff199b2-1ad0-64f2-0a0c-488f90c30f72 Address:cff199b2-1ad0-64f2-0a0c-488f90c30f72} {Suffrage:Voter ID:server-5afe6099-022d-d30b-6dd8-a6cdda2262f0 Address:5afe6099-022d-d30b-6dd8-a6cdda2262f0} {Suffrage:Voter ID:server-768b639f-e5de-e062-fbcf-7f097b2c5398 Address:768b639f-e5de-e062-fbcf-7f097b2c5398}]"
2024-05-07T18:34:55.095Z [INFO]  server-768b639f-e5de-e062-fbcf-7f097b2c5398: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cff199b2-1ad0-64f2-0a0c-488f90c30f72 Address:cff199b2-1ad0-64f2-0a0c-488f90c30f72} {Suffrage:Voter ID:server-5afe6099-022d-d30b-6dd8-a6cdda2262f0 Address:5afe6099-022d-d30b-6dd8-a6cdda2262f0} {Suffrage:Voter ID:server-768b639f-e5de-e062-fbcf-7f097b2c5398 Address:768b639f-e5de-e062-fbcf-7f097b2c5398}]"
2024-05-07T18:34:55.095Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:55.095Z [INFO]  server-cff199b2-1ad0-64f2-0a0c-488f90c30f72: entering follower state: follower="Node at cff199b2-1ad0-64f2-0a0c-488f90c30f72 [Follower]" leader-address= leader-id=
2024-05-07T18:34:55.095Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: entering follower state: follower="Node at 5afe6099-022d-d30b-6dd8-a6cdda2262f0 [Follower]" leader-address= leader-id=
2024-05-07T18:34:55.095Z [INFO]  server-768b639f-e5de-e062-fbcf-7f097b2c5398: entering follower state: follower="Node at 768b639f-e5de-e062-fbcf-7f097b2c5398 [Follower]" leader-address= leader-id=
2024-05-07T18:34:55.637Z [WARN]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:55.637Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: entering candidate state: node="Node at 5afe6099-022d-d30b-6dd8-a6cdda2262f0 [Candidate]" term=2
2024-05-07T18:34:55.639Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: election won: term=2 tally=2
2024-05-07T18:34:55.639Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: entering leader state: leader="Node at 5afe6099-022d-d30b-6dd8-a6cdda2262f0 [Leader]"
2024-05-07T18:34:55.639Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: added peer, starting replication: peer=server-cff199b2-1ad0-64f2-0a0c-488f90c30f72
2024-05-07T18:34:55.639Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: added peer, starting replication: peer=server-768b639f-e5de-e062-fbcf-7f097b2c5398
2024-05-07T18:34:55.639Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: pipelining replication: peer="{Voter server-cff199b2-1ad0-64f2-0a0c-488f90c30f72 cff199b2-1ad0-64f2-0a0c-488f90c30f72}"
2024-05-07T18:34:55.639Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: pipelining replication: peer="{Voter server-768b639f-e5de-e062-fbcf-7f097b2c5398 768b639f-e5de-e062-fbcf-7f097b2c5398}"
2024-05-07T18:34:56.648Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:55.643180556 +0000 UTC m=+33.244298806 (1 nodes), 547.553268ms from start of poll, 547.758881ms from cluster start. Timeout at 2024-05-07 18:34:56.648711199 +0000 UTC m=+34.249829440, 1.005530634s after stability
2024-05-07T18:34:56.649Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: starting snapshot up to: index=12
2024-05-07T18:34:56.649Z [INFO]  creating new snapshot: path=/tmp/raft824715490/snapshots/2-12-1715106896649.tmp
2024-05-07T18:34:56.661Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: snapshot complete up to: index=12
2024-05-07T18:34:56.661Z [INFO]  creating new snapshot: path=/tmp/raft824715490/snapshots/2-23-1715106896661.tmp
2024-05-07T18:34:56.661Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: copied to local snapshot: bytes=71
2024-05-07T18:34:56.662Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: snapshot restore progress: id=2-23-1715106896661 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:56.662Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: removing all old logs from log store
2024-05-07T18:34:56.662Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: compacting logs: from=1 to=22
2024-05-07T18:34:56.662Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: restored user snapshot: index=23
2024-05-07T18:34:56.662Z [ERROR] server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: failed to get log: index=22 error="log not found"
2024-05-07T18:34:56.662Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: aborting pipeline replication: peer="{Voter server-768b639f-e5de-e062-fbcf-7f097b2c5398 768b639f-e5de-e062-fbcf-7f097b2c5398}"
2024-05-07T18:34:56.662Z [ERROR] server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: failed to get log: index=22 error="log not found"
2024-05-07T18:34:56.662Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: aborting pipeline replication: peer="{Voter server-cff199b2-1ad0-64f2-0a0c-488f90c30f72 cff199b2-1ad0-64f2-0a0c-488f90c30f72}"
2024-05-07T18:34:56.674Z [ERROR] server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: failed to get log: index=22 error="log not found"
2024-05-07T18:34:56.674Z [ERROR] server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: failed to get log: index=22 error="log not found"
2024-05-07T18:34:56.809Z [INFO]  creating new snapshot: path=/tmp/raft3394320850/snapshots/2-23-1715106896809.tmp
2024-05-07T18:34:56.809Z [INFO]  server-cff199b2-1ad0-64f2-0a0c-488f90c30f72: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:56.809Z [INFO]  creating new snapshot: path=/tmp/raft189647759/snapshots/2-23-1715106896809.tmp
2024-05-07T18:34:56.810Z [INFO]  server-768b639f-e5de-e062-fbcf-7f097b2c5398: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:56.869Z [INFO]  server-768b639f-e5de-e062-fbcf-7f097b2c5398: copied to local snapshot: bytes=71
2024-05-07T18:34:56.869Z [INFO]  server-768b639f-e5de-e062-fbcf-7f097b2c5398: snapshot restore progress: id=2-23-1715106896809 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:56.869Z [INFO]  server-768b639f-e5de-e062-fbcf-7f097b2c5398: removing all old logs from log store
2024-05-07T18:34:56.869Z [INFO]  server-768b639f-e5de-e062-fbcf-7f097b2c5398: compacting logs: from=1 to=22
2024-05-07T18:34:56.869Z [INFO]  server-768b639f-e5de-e062-fbcf-7f097b2c5398: Installed remote snapshot
2024-05-07T18:34:56.869Z [WARN]  server-768b639f-e5de-e062-fbcf-7f097b2c5398: skipping application of old log: index=22
2024-05-07T18:34:56.869Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: pipelining replication: peer="{Voter server-768b639f-e5de-e062-fbcf-7f097b2c5398 768b639f-e5de-e062-fbcf-7f097b2c5398}"
2024-05-07T18:34:56.870Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:34:56.870Z [INFO]  server-cff199b2-1ad0-64f2-0a0c-488f90c30f72: copied to local snapshot: bytes=71
2024-05-07T18:34:56.870Z [INFO]  server-cff199b2-1ad0-64f2-0a0c-488f90c30f72: snapshot restore progress: id=2-23-1715106896809 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:34:56.870Z [INFO]  server-cff199b2-1ad0-64f2-0a0c-488f90c30f72: removing all old logs from log store
2024-05-07T18:34:56.870Z [INFO]  server-cff199b2-1ad0-64f2-0a0c-488f90c30f72: compacting logs: from=1 to=22
2024-05-07T18:34:56.870Z [INFO]  server-cff199b2-1ad0-64f2-0a0c-488f90c30f72: Installed remote snapshot
2024-05-07T18:34:56.870Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: pipelining replication: peer="{Voter server-cff199b2-1ad0-64f2-0a0c-488f90c30f72 cff199b2-1ad0-64f2-0a0c-488f90c30f72}"
2024-05-07T18:34:57.878Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:34:56.870037605 +0000 UTC m=+34.471155855 (2 nodes), 631ns from start of poll, 1.77461593s from cluster start. Timeout at 2024-05-07 18:34:57.878480123 +0000 UTC m=+35.479598363, 1.008442508s after stability
2024-05-07T18:34:57.891Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: aborting pipeline replication: peer="{Voter server-768b639f-e5de-e062-fbcf-7f097b2c5398 768b639f-e5de-e062-fbcf-7f097b2c5398}"
2024-05-07T18:34:57.891Z [INFO]  server-5afe6099-022d-d30b-6dd8-a6cdda2262f0: aborting pipeline replication: peer="{Voter server-cff199b2-1ad0-64f2-0a0c-488f90c30f72 cff199b2-1ad0-64f2-0a0c-488f90c30f72}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2024-05-07T18:34:57.892Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab03018f-3208-2345-59de-ec441c52e4af Address:ab03018f-3208-2345-59de-ec441c52e4af} {Suffrage:Voter ID:server-19996e9f-197c-4f69-c4c2-7351a83a1b7f Address:19996e9f-197c-4f69-c4c2-7351a83a1b7f} {Suffrage:Voter ID:server-1149a942-f6b5-06dc-67a5-040294db3919 Address:1149a942-f6b5-06dc-67a5-040294db3919}]"
2024-05-07T18:34:57.892Z [INFO]  server-19996e9f-197c-4f69-c4c2-7351a83a1b7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab03018f-3208-2345-59de-ec441c52e4af Address:ab03018f-3208-2345-59de-ec441c52e4af} {Suffrage:Voter ID:server-19996e9f-197c-4f69-c4c2-7351a83a1b7f Address:19996e9f-197c-4f69-c4c2-7351a83a1b7f} {Suffrage:Voter ID:server-1149a942-f6b5-06dc-67a5-040294db3919 Address:1149a942-f6b5-06dc-67a5-040294db3919}]"
2024-05-07T18:34:57.892Z [INFO]  server-1149a942-f6b5-06dc-67a5-040294db3919: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab03018f-3208-2345-59de-ec441c52e4af Address:ab03018f-3208-2345-59de-ec441c52e4af} {Suffrage:Voter ID:server-19996e9f-197c-4f69-c4c2-7351a83a1b7f Address:19996e9f-197c-4f69-c4c2-7351a83a1b7f} {Suffrage:Voter ID:server-1149a942-f6b5-06dc-67a5-040294db3919 Address:1149a942-f6b5-06dc-67a5-040294db3919}]"
2024-05-07T18:34:57.892Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:57.892Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: entering follower state: follower="Node at ab03018f-3208-2345-59de-ec441c52e4af [Follower]" leader-address= leader-id=
2024-05-07T18:34:57.892Z [INFO]  server-19996e9f-197c-4f69-c4c2-7351a83a1b7f: entering follower state: follower="Node at 19996e9f-197c-4f69-c4c2-7351a83a1b7f [Follower]" leader-address= leader-id=
2024-05-07T18:34:57.892Z [INFO]  server-1149a942-f6b5-06dc-67a5-040294db3919: entering follower state: follower="Node at 1149a942-f6b5-06dc-67a5-040294db3919 [Follower]" leader-address= leader-id=
2024-05-07T18:34:58.483Z [WARN]  server-ab03018f-3208-2345-59de-ec441c52e4af: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:34:58.483Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: entering candidate state: node="Node at ab03018f-3208-2345-59de-ec441c52e4af [Candidate]" term=2
2024-05-07T18:34:58.483Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: election won: term=2 tally=2
2024-05-07T18:34:58.483Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: entering leader state: leader="Node at ab03018f-3208-2345-59de-ec441c52e4af [Leader]"
2024-05-07T18:34:58.483Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: added peer, starting replication: peer=server-19996e9f-197c-4f69-c4c2-7351a83a1b7f
2024-05-07T18:34:58.483Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: added peer, starting replication: peer=server-1149a942-f6b5-06dc-67a5-040294db3919
2024-05-07T18:34:58.483Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: pipelining replication: peer="{Voter server-1149a942-f6b5-06dc-67a5-040294db3919 1149a942-f6b5-06dc-67a5-040294db3919}"
2024-05-07T18:34:58.483Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: pipelining replication: peer="{Voter server-19996e9f-197c-4f69-c4c2-7351a83a1b7f 19996e9f-197c-4f69-c4c2-7351a83a1b7f}"
2024-05-07T18:34:59.488Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:34:58.483696352 +0000 UTC m=+36.084814612 (1 nodes), 591.139979ms from start of poll, 591.378622ms from cluster start. Timeout at 2024-05-07 18:34:59.488910645 +0000 UTC m=+37.090028905, 1.005214293s after stability
2024-05-07T18:34:59.489Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: starting snapshot up to: index=12
2024-05-07T18:34:59.489Z [INFO]  creating new snapshot: path=/tmp/raft4209154092/snapshots/2-12-1715106899489.tmp
2024-05-07T18:34:59.505Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: snapshot complete up to: index=12
2024-05-07T18:34:59.507Z [INFO]  server-98d668db-ba39-a93c-958c-76802d7d7d7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98d668db-ba39-a93c-958c-76802d7d7d7f Address:98d668db-ba39-a93c-958c-76802d7d7d7f} {Suffrage:Voter ID:server-3af0c833-b4af-0ab1-bdc3-66a598bb4235 Address:3af0c833-b4af-0ab1-bdc3-66a598bb4235} {Suffrage:Voter ID:server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa Address:df13fd26-dc05-7ef9-281e-b4cab7b4e5fa}]"
2024-05-07T18:34:59.507Z [INFO]  server-3af0c833-b4af-0ab1-bdc3-66a598bb4235: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98d668db-ba39-a93c-958c-76802d7d7d7f Address:98d668db-ba39-a93c-958c-76802d7d7d7f} {Suffrage:Voter ID:server-3af0c833-b4af-0ab1-bdc3-66a598bb4235 Address:3af0c833-b4af-0ab1-bdc3-66a598bb4235} {Suffrage:Voter ID:server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa Address:df13fd26-dc05-7ef9-281e-b4cab7b4e5fa}]"
2024-05-07T18:34:59.507Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98d668db-ba39-a93c-958c-76802d7d7d7f Address:98d668db-ba39-a93c-958c-76802d7d7d7f} {Suffrage:Voter ID:server-3af0c833-b4af-0ab1-bdc3-66a598bb4235 Address:3af0c833-b4af-0ab1-bdc3-66a598bb4235} {Suffrage:Voter ID:server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa Address:df13fd26-dc05-7ef9-281e-b4cab7b4e5fa}]"
2024-05-07T18:34:59.507Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:34:59.507Z [INFO]  server-98d668db-ba39-a93c-958c-76802d7d7d7f: entering follower state: follower="Node at 98d668db-ba39-a93c-958c-76802d7d7d7f [Follower]" leader-address= leader-id=
2024-05-07T18:34:59.507Z [INFO]  server-3af0c833-b4af-0ab1-bdc3-66a598bb4235: entering follower state: follower="Node at 3af0c833-b4af-0ab1-bdc3-66a598bb4235 [Follower]" leader-address= leader-id=
2024-05-07T18:34:59.507Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: entering follower state: follower="Node at df13fd26-dc05-7ef9-281e-b4cab7b4e5fa [Follower]" leader-address= leader-id=
2024-05-07T18:35:00.204Z [WARN]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:00.205Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: entering candidate state: node="Node at df13fd26-dc05-7ef9-281e-b4cab7b4e5fa [Candidate]" term=2
2024-05-07T18:35:00.205Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: election won: term=2 tally=2
2024-05-07T18:35:00.205Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: entering leader state: leader="Node at df13fd26-dc05-7ef9-281e-b4cab7b4e5fa [Leader]"
2024-05-07T18:35:00.205Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: added peer, starting replication: peer=server-98d668db-ba39-a93c-958c-76802d7d7d7f
2024-05-07T18:35:00.205Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: added peer, starting replication: peer=server-3af0c833-b4af-0ab1-bdc3-66a598bb4235
2024-05-07T18:35:00.205Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: pipelining replication: peer="{Voter server-3af0c833-b4af-0ab1-bdc3-66a598bb4235 3af0c833-b4af-0ab1-bdc3-66a598bb4235}"
2024-05-07T18:35:00.205Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: pipelining replication: peer="{Voter server-98d668db-ba39-a93c-958c-76802d7d7d7f 98d668db-ba39-a93c-958c-76802d7d7d7f}"
2024-05-07T18:35:01.211Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:00.205220483 +0000 UTC m=+37.806338733 (1 nodes), 697.699934ms from start of poll, 697.985285ms from cluster start. Timeout at 2024-05-07 18:35:01.210957368 +0000 UTC m=+38.812075618, 1.005736885s after stability
2024-05-07T18:35:01.211Z [INFO]  creating new snapshot: path=/tmp/raft3628460643/snapshots/2-14-1715106901211.tmp
2024-05-07T18:35:01.211Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: copied to local snapshot: bytes=71
2024-05-07T18:35:01.212Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: snapshot restore progress: id=2-14-1715106901211 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:01.212Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: restored user snapshot: index=14
2024-05-07T18:35:01.212Z [ERROR] server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: failed to get log: index=3 error="log not found"
2024-05-07T18:35:01.212Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: aborting pipeline replication: peer="{Voter server-3af0c833-b4af-0ab1-bdc3-66a598bb4235 3af0c833-b4af-0ab1-bdc3-66a598bb4235}"
2024-05-07T18:35:01.212Z [ERROR] server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: failed to get log: index=3 error="log not found"
2024-05-07T18:35:01.212Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: aborting pipeline replication: peer="{Voter server-98d668db-ba39-a93c-958c-76802d7d7d7f 98d668db-ba39-a93c-958c-76802d7d7d7f}"
2024-05-07T18:35:01.220Z [ERROR] server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: failed to get log: index=3 error="log not found"
2024-05-07T18:35:01.220Z [INFO]  creating new snapshot: path=/tmp/raft3814174890/snapshots/2-14-1715106901220.tmp
2024-05-07T18:35:01.220Z [ERROR] server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: failed to get log: index=3 error="log not found"
2024-05-07T18:35:01.220Z [INFO]  server-98d668db-ba39-a93c-958c-76802d7d7d7f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:01.221Z [INFO]  creating new snapshot: path=/tmp/raft875610657/snapshots/2-14-1715106901221.tmp
2024-05-07T18:35:01.221Z [INFO]  server-98d668db-ba39-a93c-958c-76802d7d7d7f: copied to local snapshot: bytes=71
2024-05-07T18:35:01.221Z [INFO]  server-3af0c833-b4af-0ab1-bdc3-66a598bb4235: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:01.221Z [INFO]  server-98d668db-ba39-a93c-958c-76802d7d7d7f: snapshot restore progress: id=2-14-1715106901220 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:01.221Z [INFO]  server-3af0c833-b4af-0ab1-bdc3-66a598bb4235: copied to local snapshot: bytes=71
2024-05-07T18:35:01.221Z [INFO]  server-98d668db-ba39-a93c-958c-76802d7d7d7f: Installed remote snapshot
2024-05-07T18:35:01.222Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: pipelining replication: peer="{Voter server-98d668db-ba39-a93c-958c-76802d7d7d7f 98d668db-ba39-a93c-958c-76802d7d7d7f}"
2024-05-07T18:35:01.222Z [INFO]  server-3af0c833-b4af-0ab1-bdc3-66a598bb4235: snapshot restore progress: id=2-14-1715106901221 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:01.222Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:01.222Z [INFO]  server-3af0c833-b4af-0ab1-bdc3-66a598bb4235: Installed remote snapshot
2024-05-07T18:35:01.222Z [INFO]  server-df13fd26-dc05-7ef9-281e-b4cab7b4e5fa: pipelining replication: peer="{Voter server-3af0c833-b4af-0ab1-bdc3-66a598bb4235 3af0c833-b4af-0ab1-bdc3-66a598bb4235}"
2024-05-07T18:35:02.227Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:01.222158452 +0000 UTC m=+38.823276712 (2 nodes), 1.192µs from start of poll, 1.714923264s from cluster start. Timeout at 2024-05-07 18:35:02.227184875 +0000 UTC m=+39.828303125, 1.005026413s after stability
2024-05-07T18:35:02.241Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: aborting pipeline replication: peer="{Voter server-1149a942-f6b5-06dc-67a5-040294db3919 1149a942-f6b5-06dc-67a5-040294db3919}"
2024-05-07T18:35:02.241Z [INFO]  server-ab03018f-3208-2345-59de-ec441c52e4af: aborting pipeline replication: peer="{Voter server-19996e9f-197c-4f69-c4c2-7351a83a1b7f 19996e9f-197c-4f69-c4c2-7351a83a1b7f}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2024-05-07T18:35:02.244Z [INFO]  server-535e0d5c-d4cd-810d-b855-d4a61b01a130: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-535e0d5c-d4cd-810d-b855-d4a61b01a130 Address:535e0d5c-d4cd-810d-b855-d4a61b01a130} {Suffrage:Voter ID:server-ede357ff-be4c-8e9a-bed4-19b7f382f57e Address:ede357ff-be4c-8e9a-bed4-19b7f382f57e} {Suffrage:Voter ID:server-a87ec69c-be58-c297-36b9-2a41679c8d7e Address:a87ec69c-be58-c297-36b9-2a41679c8d7e}]"
2024-05-07T18:35:02.244Z [INFO]  server-ede357ff-be4c-8e9a-bed4-19b7f382f57e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-535e0d5c-d4cd-810d-b855-d4a61b01a130 Address:535e0d5c-d4cd-810d-b855-d4a61b01a130} {Suffrage:Voter ID:server-ede357ff-be4c-8e9a-bed4-19b7f382f57e Address:ede357ff-be4c-8e9a-bed4-19b7f382f57e} {Suffrage:Voter ID:server-a87ec69c-be58-c297-36b9-2a41679c8d7e Address:a87ec69c-be58-c297-36b9-2a41679c8d7e}]"
2024-05-07T18:35:02.244Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-535e0d5c-d4cd-810d-b855-d4a61b01a130 Address:535e0d5c-d4cd-810d-b855-d4a61b01a130} {Suffrage:Voter ID:server-ede357ff-be4c-8e9a-bed4-19b7f382f57e Address:ede357ff-be4c-8e9a-bed4-19b7f382f57e} {Suffrage:Voter ID:server-a87ec69c-be58-c297-36b9-2a41679c8d7e Address:a87ec69c-be58-c297-36b9-2a41679c8d7e}]"
2024-05-07T18:35:02.244Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:02.244Z [INFO]  server-535e0d5c-d4cd-810d-b855-d4a61b01a130: entering follower state: follower="Node at 535e0d5c-d4cd-810d-b855-d4a61b01a130 [Follower]" leader-address= leader-id=
2024-05-07T18:35:02.244Z [INFO]  server-ede357ff-be4c-8e9a-bed4-19b7f382f57e: entering follower state: follower="Node at ede357ff-be4c-8e9a-bed4-19b7f382f57e [Follower]" leader-address= leader-id=
2024-05-07T18:35:02.244Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: entering follower state: follower="Node at a87ec69c-be58-c297-36b9-2a41679c8d7e [Follower]" leader-address= leader-id=
2024-05-07T18:35:02.777Z [WARN]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:02.777Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: entering candidate state: node="Node at a87ec69c-be58-c297-36b9-2a41679c8d7e [Candidate]" term=2
2024-05-07T18:35:02.777Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: election won: term=2 tally=2
2024-05-07T18:35:02.777Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: entering leader state: leader="Node at a87ec69c-be58-c297-36b9-2a41679c8d7e [Leader]"
2024-05-07T18:35:02.777Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: added peer, starting replication: peer=server-535e0d5c-d4cd-810d-b855-d4a61b01a130
2024-05-07T18:35:02.777Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: added peer, starting replication: peer=server-ede357ff-be4c-8e9a-bed4-19b7f382f57e
2024-05-07T18:35:02.779Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: pipelining replication: peer="{Voter server-535e0d5c-d4cd-810d-b855-d4a61b01a130 535e0d5c-d4cd-810d-b855-d4a61b01a130}"
2024-05-07T18:35:02.779Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: pipelining replication: peer="{Voter server-ede357ff-be4c-8e9a-bed4-19b7f382f57e ede357ff-be4c-8e9a-bed4-19b7f382f57e}"
2024-05-07T18:35:03.785Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:02.779678864 +0000 UTC m=+40.380797114 (1 nodes), 534.928206ms from start of poll, 535.449476ms from cluster start. Timeout at 2024-05-07 18:35:03.784952268 +0000 UTC m=+41.386070518, 1.005273404s after stability
2024-05-07T18:35:03.785Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: starting snapshot up to: index=12
2024-05-07T18:35:03.785Z [INFO]  creating new snapshot: path=/tmp/raft1703772654/snapshots/2-12-1715106903785.tmp
2024-05-07T18:35:03.825Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: snapshot complete up to: index=12
2024-05-07T18:35:03.827Z [INFO]  server-6e47f1ec-c216-da56-95f7-a54484455bc8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e47f1ec-c216-da56-95f7-a54484455bc8 Address:6e47f1ec-c216-da56-95f7-a54484455bc8} {Suffrage:Voter ID:server-4d44f159-a92a-9d4f-4e60-1261928033bc Address:4d44f159-a92a-9d4f-4e60-1261928033bc} {Suffrage:Voter ID:server-736fdbe0-d885-c4df-ff27-51bd2990185a Address:736fdbe0-d885-c4df-ff27-51bd2990185a}]"
2024-05-07T18:35:03.827Z [INFO]  server-6e47f1ec-c216-da56-95f7-a54484455bc8: entering follower state: follower="Node at 6e47f1ec-c216-da56-95f7-a54484455bc8 [Follower]" leader-address= leader-id=
2024-05-07T18:35:03.827Z [INFO]  server-4d44f159-a92a-9d4f-4e60-1261928033bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e47f1ec-c216-da56-95f7-a54484455bc8 Address:6e47f1ec-c216-da56-95f7-a54484455bc8} {Suffrage:Voter ID:server-4d44f159-a92a-9d4f-4e60-1261928033bc Address:4d44f159-a92a-9d4f-4e60-1261928033bc} {Suffrage:Voter ID:server-736fdbe0-d885-c4df-ff27-51bd2990185a Address:736fdbe0-d885-c4df-ff27-51bd2990185a}]"
2024-05-07T18:35:03.827Z [INFO]  server-4d44f159-a92a-9d4f-4e60-1261928033bc: entering follower state: follower="Node at 4d44f159-a92a-9d4f-4e60-1261928033bc [Follower]" leader-address= leader-id=
2024-05-07T18:35:03.827Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e47f1ec-c216-da56-95f7-a54484455bc8 Address:6e47f1ec-c216-da56-95f7-a54484455bc8} {Suffrage:Voter ID:server-4d44f159-a92a-9d4f-4e60-1261928033bc Address:4d44f159-a92a-9d4f-4e60-1261928033bc} {Suffrage:Voter ID:server-736fdbe0-d885-c4df-ff27-51bd2990185a Address:736fdbe0-d885-c4df-ff27-51bd2990185a}]"
2024-05-07T18:35:03.827Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:03.827Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: entering follower state: follower="Node at 736fdbe0-d885-c4df-ff27-51bd2990185a [Follower]" leader-address= leader-id=
2024-05-07T18:35:04.436Z [WARN]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:04.436Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: entering candidate state: node="Node at 736fdbe0-d885-c4df-ff27-51bd2990185a [Candidate]" term=2
2024-05-07T18:35:04.436Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: election won: term=2 tally=2
2024-05-07T18:35:04.436Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: entering leader state: leader="Node at 736fdbe0-d885-c4df-ff27-51bd2990185a [Leader]"
2024-05-07T18:35:04.436Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: added peer, starting replication: peer=server-6e47f1ec-c216-da56-95f7-a54484455bc8
2024-05-07T18:35:04.436Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: added peer, starting replication: peer=server-4d44f159-a92a-9d4f-4e60-1261928033bc
2024-05-07T18:35:04.436Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: pipelining replication: peer="{Voter server-4d44f159-a92a-9d4f-4e60-1261928033bc 4d44f159-a92a-9d4f-4e60-1261928033bc}"
2024-05-07T18:35:04.436Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: pipelining replication: peer="{Voter server-6e47f1ec-c216-da56-95f7-a54484455bc8 6e47f1ec-c216-da56-95f7-a54484455bc8}"
2024-05-07T18:35:05.441Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:04.43642982 +0000 UTC m=+42.037548070 (1 nodes), 608.721879ms from start of poll, 609.327094ms from cluster start. Timeout at 2024-05-07 18:35:05.441479268 +0000 UTC m=+43.042597518, 1.005049448s after stability
2024-05-07T18:35:05.501Z [INFO]  creating new snapshot: path=/tmp/raft1625847262/snapshots/2-14-1715106905501.tmp
2024-05-07T18:35:05.602Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: copied to local snapshot: bytes=71
2024-05-07T18:35:05.602Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: snapshot restore progress: id=2-14-1715106905501 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:05.602Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: removing all old logs from log store
2024-05-07T18:35:05.602Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: compacting logs: from=1 to=2
2024-05-07T18:35:05.602Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: restored user snapshot: index=14
2024-05-07T18:35:05.602Z [ERROR] server-736fdbe0-d885-c4df-ff27-51bd2990185a: failed to get log: index=2 error="log not found"
2024-05-07T18:35:05.602Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: aborting pipeline replication: peer="{Voter server-4d44f159-a92a-9d4f-4e60-1261928033bc 4d44f159-a92a-9d4f-4e60-1261928033bc}"
2024-05-07T18:35:05.602Z [ERROR] server-736fdbe0-d885-c4df-ff27-51bd2990185a: failed to get log: index=2 error="log not found"
2024-05-07T18:35:05.602Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: aborting pipeline replication: peer="{Voter server-6e47f1ec-c216-da56-95f7-a54484455bc8 6e47f1ec-c216-da56-95f7-a54484455bc8}"
2024-05-07T18:35:05.608Z [ERROR] server-736fdbe0-d885-c4df-ff27-51bd2990185a: failed to get log: index=2 error="log not found"
2024-05-07T18:35:05.608Z [ERROR] server-736fdbe0-d885-c4df-ff27-51bd2990185a: failed to get log: index=2 error="log not found"
2024-05-07T18:35:05.608Z [INFO]  creating new snapshot: path=/tmp/raft2414016603/snapshots/2-14-1715106905608.tmp
2024-05-07T18:35:05.609Z [INFO]  creating new snapshot: path=/tmp/raft2750590954/snapshots/2-14-1715106905609.tmp
2024-05-07T18:35:05.634Z [INFO]  server-6e47f1ec-c216-da56-95f7-a54484455bc8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:05.634Z [INFO]  server-4d44f159-a92a-9d4f-4e60-1261928033bc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:05.652Z [INFO]  server-6e47f1ec-c216-da56-95f7-a54484455bc8: copied to local snapshot: bytes=71
2024-05-07T18:35:05.652Z [INFO]  server-6e47f1ec-c216-da56-95f7-a54484455bc8: snapshot restore progress: id=2-14-1715106905608 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:05.652Z [INFO]  server-6e47f1ec-c216-da56-95f7-a54484455bc8: removing all old logs from log store
2024-05-07T18:35:05.652Z [INFO]  server-6e47f1ec-c216-da56-95f7-a54484455bc8: compacting logs: from=1 to=2
2024-05-07T18:35:05.652Z [INFO]  server-6e47f1ec-c216-da56-95f7-a54484455bc8: Installed remote snapshot
2024-05-07T18:35:05.652Z [INFO]  server-4d44f159-a92a-9d4f-4e60-1261928033bc: copied to local snapshot: bytes=71
2024-05-07T18:35:05.653Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: pipelining replication: peer="{Voter server-6e47f1ec-c216-da56-95f7-a54484455bc8 6e47f1ec-c216-da56-95f7-a54484455bc8}"
2024-05-07T18:35:05.653Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:05.653Z [INFO]  server-4d44f159-a92a-9d4f-4e60-1261928033bc: snapshot restore progress: id=2-14-1715106905609 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:05.653Z [INFO]  server-4d44f159-a92a-9d4f-4e60-1261928033bc: removing all old logs from log store
2024-05-07T18:35:05.653Z [INFO]  server-4d44f159-a92a-9d4f-4e60-1261928033bc: compacting logs: from=1 to=2
2024-05-07T18:35:05.653Z [INFO]  server-4d44f159-a92a-9d4f-4e60-1261928033bc: Installed remote snapshot
2024-05-07T18:35:05.653Z [INFO]  server-736fdbe0-d885-c4df-ff27-51bd2990185a: pipelining replication: peer="{Voter server-4d44f159-a92a-9d4f-4e60-1261928033bc 4d44f159-a92a-9d4f-4e60-1261928033bc}"
2024-05-07T18:35:06.658Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:05.653147529 +0000 UTC m=+43.254265779 (2 nodes), 1.393µs from start of poll, 1.826044803s from cluster start. Timeout at 2024-05-07 18:35:06.658610345 +0000 UTC m=+44.259728595, 1.005462816s after stability
2024-05-07T18:35:06.670Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: aborting pipeline replication: peer="{Voter server-ede357ff-be4c-8e9a-bed4-19b7f382f57e ede357ff-be4c-8e9a-bed4-19b7f382f57e}"
2024-05-07T18:35:06.670Z [INFO]  server-a87ec69c-be58-c297-36b9-2a41679c8d7e: aborting pipeline replication: peer="{Voter server-535e0d5c-d4cd-810d-b855-d4a61b01a130 535e0d5c-d4cd-810d-b855-d4a61b01a130}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2024-05-07T18:35:06.672Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6dcb2efd-fe8c-cf79-a719-4b2978527767 Address:6dcb2efd-fe8c-cf79-a719-4b2978527767} {Suffrage:Voter ID:server-86baf8ca-e923-ba3d-0003-4e987aa478c1 Address:86baf8ca-e923-ba3d-0003-4e987aa478c1} {Suffrage:Voter ID:server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815 Address:6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815}]"
2024-05-07T18:35:06.672Z [INFO]  server-86baf8ca-e923-ba3d-0003-4e987aa478c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6dcb2efd-fe8c-cf79-a719-4b2978527767 Address:6dcb2efd-fe8c-cf79-a719-4b2978527767} {Suffrage:Voter ID:server-86baf8ca-e923-ba3d-0003-4e987aa478c1 Address:86baf8ca-e923-ba3d-0003-4e987aa478c1} {Suffrage:Voter ID:server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815 Address:6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815}]"
2024-05-07T18:35:06.672Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: entering follower state: follower="Node at 6dcb2efd-fe8c-cf79-a719-4b2978527767 [Follower]" leader-address= leader-id=
2024-05-07T18:35:06.672Z [INFO]  server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6dcb2efd-fe8c-cf79-a719-4b2978527767 Address:6dcb2efd-fe8c-cf79-a719-4b2978527767} {Suffrage:Voter ID:server-86baf8ca-e923-ba3d-0003-4e987aa478c1 Address:86baf8ca-e923-ba3d-0003-4e987aa478c1} {Suffrage:Voter ID:server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815 Address:6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815}]"
2024-05-07T18:35:06.672Z [INFO]  server-86baf8ca-e923-ba3d-0003-4e987aa478c1: entering follower state: follower="Node at 86baf8ca-e923-ba3d-0003-4e987aa478c1 [Follower]" leader-address= leader-id=
2024-05-07T18:35:06.672Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:06.672Z [INFO]  server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815: entering follower state: follower="Node at 6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815 [Follower]" leader-address= leader-id=
2024-05-07T18:35:07.178Z [WARN]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:07.178Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: entering candidate state: node="Node at 6dcb2efd-fe8c-cf79-a719-4b2978527767 [Candidate]" term=2
2024-05-07T18:35:07.178Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: election won: term=2 tally=2
2024-05-07T18:35:07.178Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: entering leader state: leader="Node at 6dcb2efd-fe8c-cf79-a719-4b2978527767 [Leader]"
2024-05-07T18:35:07.178Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: added peer, starting replication: peer=server-86baf8ca-e923-ba3d-0003-4e987aa478c1
2024-05-07T18:35:07.178Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: added peer, starting replication: peer=server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815
2024-05-07T18:35:07.178Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: pipelining replication: peer="{Voter server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815 6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815}"
2024-05-07T18:35:07.178Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: pipelining replication: peer="{Voter server-86baf8ca-e923-ba3d-0003-4e987aa478c1 86baf8ca-e923-ba3d-0003-4e987aa478c1}"
2024-05-07T18:35:08.184Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:07.178771116 +0000 UTC m=+44.779889366 (1 nodes), 506.382154ms from start of poll, 506.744708ms from cluster start. Timeout at 2024-05-07 18:35:08.184427762 +0000 UTC m=+45.785546012, 1.005656646s after stability
2024-05-07T18:35:08.185Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: starting snapshot up to: index=12
2024-05-07T18:35:08.185Z [INFO]  creating new snapshot: path=/tmp/raft3089221544/snapshots/2-12-1715106908185.tmp
2024-05-07T18:35:08.209Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: snapshot complete up to: index=12
2024-05-07T18:35:08.210Z [INFO]  creating new snapshot: path=/tmp/raft3089221544/snapshots/2-23-1715106908210.tmp
2024-05-07T18:35:08.211Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: copied to local snapshot: bytes=71
2024-05-07T18:35:08.211Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: snapshot restore progress: id=2-23-1715106908210 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:08.211Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: restored user snapshot: index=23
2024-05-07T18:35:08.211Z [ERROR] server-6dcb2efd-fe8c-cf79-a719-4b2978527767: failed to get log: index=23 error="log not found"
2024-05-07T18:35:08.211Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: aborting pipeline replication: peer="{Voter server-86baf8ca-e923-ba3d-0003-4e987aa478c1 86baf8ca-e923-ba3d-0003-4e987aa478c1}"
2024-05-07T18:35:08.211Z [ERROR] server-6dcb2efd-fe8c-cf79-a719-4b2978527767: failed to get log: index=23 error="log not found"
2024-05-07T18:35:08.211Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: aborting pipeline replication: peer="{Voter server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815 6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815}"
2024-05-07T18:35:08.217Z [ERROR] server-6dcb2efd-fe8c-cf79-a719-4b2978527767: failed to get log: index=23 error="log not found"
2024-05-07T18:35:08.221Z [ERROR] server-6dcb2efd-fe8c-cf79-a719-4b2978527767: failed to get log: index=23 error="log not found"
2024-05-07T18:35:08.264Z [INFO]  creating new snapshot: path=/tmp/raft3476185997/snapshots/2-23-1715106908264.tmp
2024-05-07T18:35:08.266Z [INFO]  creating new snapshot: path=/tmp/raft278868016/snapshots/2-23-1715106908266.tmp
2024-05-07T18:35:08.266Z [INFO]  server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:08.266Z [INFO]  server-86baf8ca-e923-ba3d-0003-4e987aa478c1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:08.315Z [INFO]  server-86baf8ca-e923-ba3d-0003-4e987aa478c1: copied to local snapshot: bytes=71
2024-05-07T18:35:08.315Z [INFO]  server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815: copied to local snapshot: bytes=71
2024-05-07T18:35:08.315Z [INFO]  server-86baf8ca-e923-ba3d-0003-4e987aa478c1: snapshot restore progress: id=2-23-1715106908266 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:08.316Z [INFO]  server-86baf8ca-e923-ba3d-0003-4e987aa478c1: Installed remote snapshot
2024-05-07T18:35:08.316Z [INFO]  server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815: snapshot restore progress: id=2-23-1715106908264 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:08.316Z [INFO]  server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815: Installed remote snapshot
2024-05-07T18:35:08.316Z [WARN]  server-86baf8ca-e923-ba3d-0003-4e987aa478c1: skipping application of old log: index=22
2024-05-07T18:35:08.316Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: pipelining replication: peer="{Voter server-86baf8ca-e923-ba3d-0003-4e987aa478c1 86baf8ca-e923-ba3d-0003-4e987aa478c1}"
2024-05-07T18:35:08.316Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:08.316Z [WARN]  server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815: skipping application of old log: index=23
2024-05-07T18:35:08.316Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: pipelining replication: peer="{Voter server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815 6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815}"
2024-05-07T18:35:09.321Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:08.316286562 +0000 UTC m=+45.917404812 (2 nodes), 1.533µs from start of poll, 1.644260154s from cluster start. Timeout at 2024-05-07 18:35:09.321752223 +0000 UTC m=+46.922870473, 1.005465661s after stability
2024-05-07T18:35:09.333Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: aborting pipeline replication: peer="{Voter server-6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815 6e2b6f6f-4a90-c8f3-8bdf-b489a4cc8815}"
2024-05-07T18:35:09.333Z [INFO]  server-6dcb2efd-fe8c-cf79-a719-4b2978527767: aborting pipeline replication: peer="{Voter server-86baf8ca-e923-ba3d-0003-4e987aa478c1 86baf8ca-e923-ba3d-0003-4e987aa478c1}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2024-05-07T18:35:09.337Z [INFO]  server-acd82ce8-918e-92ef-2904-d63a6e8135ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acd82ce8-918e-92ef-2904-d63a6e8135ac Address:acd82ce8-918e-92ef-2904-d63a6e8135ac} {Suffrage:Voter ID:server-fce300fc-991a-28ef-2266-f8a826465bd6 Address:fce300fc-991a-28ef-2266-f8a826465bd6} {Suffrage:Voter ID:server-6d97dcff-6877-afb8-d149-7d2abde8f241 Address:6d97dcff-6877-afb8-d149-7d2abde8f241}]"
2024-05-07T18:35:09.337Z [INFO]  server-fce300fc-991a-28ef-2266-f8a826465bd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acd82ce8-918e-92ef-2904-d63a6e8135ac Address:acd82ce8-918e-92ef-2904-d63a6e8135ac} {Suffrage:Voter ID:server-fce300fc-991a-28ef-2266-f8a826465bd6 Address:fce300fc-991a-28ef-2266-f8a826465bd6} {Suffrage:Voter ID:server-6d97dcff-6877-afb8-d149-7d2abde8f241 Address:6d97dcff-6877-afb8-d149-7d2abde8f241}]"
2024-05-07T18:35:09.337Z [INFO]  server-acd82ce8-918e-92ef-2904-d63a6e8135ac: entering follower state: follower="Node at acd82ce8-918e-92ef-2904-d63a6e8135ac [Follower]" leader-address= leader-id=
2024-05-07T18:35:09.337Z [INFO]  server-fce300fc-991a-28ef-2266-f8a826465bd6: entering follower state: follower="Node at fce300fc-991a-28ef-2266-f8a826465bd6 [Follower]" leader-address= leader-id=
2024-05-07T18:35:09.337Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acd82ce8-918e-92ef-2904-d63a6e8135ac Address:acd82ce8-918e-92ef-2904-d63a6e8135ac} {Suffrage:Voter ID:server-fce300fc-991a-28ef-2266-f8a826465bd6 Address:fce300fc-991a-28ef-2266-f8a826465bd6} {Suffrage:Voter ID:server-6d97dcff-6877-afb8-d149-7d2abde8f241 Address:6d97dcff-6877-afb8-d149-7d2abde8f241}]"
2024-05-07T18:35:09.337Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:09.337Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: entering follower state: follower="Node at 6d97dcff-6877-afb8-d149-7d2abde8f241 [Follower]" leader-address= leader-id=
2024-05-07T18:35:09.843Z [WARN]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:09.843Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: entering candidate state: node="Node at 6d97dcff-6877-afb8-d149-7d2abde8f241 [Candidate]" term=2
2024-05-07T18:35:09.843Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: election won: term=2 tally=2
2024-05-07T18:35:09.843Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: entering leader state: leader="Node at 6d97dcff-6877-afb8-d149-7d2abde8f241 [Leader]"
2024-05-07T18:35:09.843Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: added peer, starting replication: peer=server-acd82ce8-918e-92ef-2904-d63a6e8135ac
2024-05-07T18:35:09.843Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: added peer, starting replication: peer=server-fce300fc-991a-28ef-2266-f8a826465bd6
2024-05-07T18:35:09.843Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: pipelining replication: peer="{Voter server-fce300fc-991a-28ef-2266-f8a826465bd6 fce300fc-991a-28ef-2266-f8a826465bd6}"
2024-05-07T18:35:09.843Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: pipelining replication: peer="{Voter server-acd82ce8-918e-92ef-2904-d63a6e8135ac acd82ce8-918e-92ef-2904-d63a6e8135ac}"
2024-05-07T18:35:10.849Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:09.843699707 +0000 UTC m=+47.444817957 (1 nodes), 506.35914ms from start of poll, 506.72421ms from cluster start. Timeout at 2024-05-07 18:35:10.84917709 +0000 UTC m=+48.450295340, 1.005477383s after stability
2024-05-07T18:35:10.852Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: starting snapshot up to: index=12
2024-05-07T18:35:10.852Z [INFO]  creating new snapshot: path=/tmp/raft1023467441/snapshots/2-12-1715106910852.tmp
2024-05-07T18:35:10.928Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: snapshot complete up to: index=12
2024-05-07T18:35:10.928Z [INFO]  creating new snapshot: path=/tmp/raft1023467441/snapshots/2-23-1715106910928.tmp
2024-05-07T18:35:11.059Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: copied to local snapshot: bytes=71
2024-05-07T18:35:11.059Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: snapshot restore progress: id=2-23-1715106910928 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:11.059Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: removing all old logs from log store
2024-05-07T18:35:11.059Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: compacting logs: from=1 to=22
2024-05-07T18:35:11.059Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: restored user snapshot: index=23
2024-05-07T18:35:11.059Z [ERROR] server-6d97dcff-6877-afb8-d149-7d2abde8f241: failed to get log: index=22 error="log not found"
2024-05-07T18:35:11.059Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: aborting pipeline replication: peer="{Voter server-fce300fc-991a-28ef-2266-f8a826465bd6 fce300fc-991a-28ef-2266-f8a826465bd6}"
2024-05-07T18:35:11.059Z [ERROR] server-6d97dcff-6877-afb8-d149-7d2abde8f241: failed to get log: index=22 error="log not found"
2024-05-07T18:35:11.059Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: aborting pipeline replication: peer="{Voter server-acd82ce8-918e-92ef-2904-d63a6e8135ac acd82ce8-918e-92ef-2904-d63a6e8135ac}"
2024-05-07T18:35:11.066Z [ERROR] server-6d97dcff-6877-afb8-d149-7d2abde8f241: failed to get log: index=22 error="log not found"
2024-05-07T18:35:11.067Z [INFO]  creating new snapshot: path=/tmp/raft3141519460/snapshots/2-23-1715106911067.tmp
2024-05-07T18:35:11.067Z [INFO]  server-fce300fc-991a-28ef-2266-f8a826465bd6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:11.068Z [ERROR] server-6d97dcff-6877-afb8-d149-7d2abde8f241: failed to get log: index=22 error="log not found"
2024-05-07T18:35:11.069Z [INFO]  creating new snapshot: path=/tmp/raft3415532260/snapshots/2-23-1715106911069.tmp
2024-05-07T18:35:11.070Z [INFO]  server-acd82ce8-918e-92ef-2904-d63a6e8135ac: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:11.250Z [INFO]  server-fce300fc-991a-28ef-2266-f8a826465bd6: copied to local snapshot: bytes=71
2024-05-07T18:35:11.250Z [INFO]  server-fce300fc-991a-28ef-2266-f8a826465bd6: snapshot restore progress: id=2-23-1715106911067 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:11.250Z [INFO]  server-fce300fc-991a-28ef-2266-f8a826465bd6: removing all old logs from log store
2024-05-07T18:35:11.250Z [INFO]  server-fce300fc-991a-28ef-2266-f8a826465bd6: compacting logs: from=1 to=22
2024-05-07T18:35:11.250Z [INFO]  server-fce300fc-991a-28ef-2266-f8a826465bd6: Installed remote snapshot
2024-05-07T18:35:11.251Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:11.251Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: pipelining replication: peer="{Voter server-fce300fc-991a-28ef-2266-f8a826465bd6 fce300fc-991a-28ef-2266-f8a826465bd6}"
2024-05-07T18:35:11.251Z [INFO]  server-acd82ce8-918e-92ef-2904-d63a6e8135ac: copied to local snapshot: bytes=71
2024-05-07T18:35:11.251Z [INFO]  server-acd82ce8-918e-92ef-2904-d63a6e8135ac: snapshot restore progress: id=2-23-1715106911069 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:11.251Z [INFO]  server-acd82ce8-918e-92ef-2904-d63a6e8135ac: removing all old logs from log store
2024-05-07T18:35:11.251Z [INFO]  server-acd82ce8-918e-92ef-2904-d63a6e8135ac: compacting logs: from=1 to=22
2024-05-07T18:35:11.251Z [INFO]  server-acd82ce8-918e-92ef-2904-d63a6e8135ac: Installed remote snapshot
2024-05-07T18:35:11.251Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: pipelining replication: peer="{Voter server-acd82ce8-918e-92ef-2904-d63a6e8135ac acd82ce8-918e-92ef-2904-d63a6e8135ac}"
2024-05-07T18:35:12.256Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:11.251126865 +0000 UTC m=+48.852245115 (2 nodes), 1.503µs from start of poll, 1.914151368s from cluster start. Timeout at 2024-05-07 18:35:12.256193083 +0000 UTC m=+49.857311333, 1.005066218s after stability
2024-05-07T18:35:12.267Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: aborting pipeline replication: peer="{Voter server-fce300fc-991a-28ef-2266-f8a826465bd6 fce300fc-991a-28ef-2266-f8a826465bd6}"
2024-05-07T18:35:12.267Z [INFO]  server-6d97dcff-6877-afb8-d149-7d2abde8f241: aborting pipeline replication: peer="{Voter server-acd82ce8-918e-92ef-2904-d63a6e8135ac acd82ce8-918e-92ef-2904-d63a6e8135ac}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2024-05-07T18:35:12.269Z [INFO]  server-77b15ebb-d061-c4f7-3c85-fd1f807a618a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77b15ebb-d061-c4f7-3c85-fd1f807a618a Address:77b15ebb-d061-c4f7-3c85-fd1f807a618a} {Suffrage:Voter ID:server-d22a04c1-dba3-4034-54ec-1bec7f3c84b9 Address:d22a04c1-dba3-4034-54ec-1bec7f3c84b9} {Suffrage:Voter ID:server-940b2c4f-327c-fce5-f457-202ee6e79a78 Address:940b2c4f-327c-fce5-f457-202ee6e79a78}]"
2024-05-07T18:35:12.269Z [INFO]  server-d22a04c1-dba3-4034-54ec-1bec7f3c84b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77b15ebb-d061-c4f7-3c85-fd1f807a618a Address:77b15ebb-d061-c4f7-3c85-fd1f807a618a} {Suffrage:Voter ID:server-d22a04c1-dba3-4034-54ec-1bec7f3c84b9 Address:d22a04c1-dba3-4034-54ec-1bec7f3c84b9} {Suffrage:Voter ID:server-940b2c4f-327c-fce5-f457-202ee6e79a78 Address:940b2c4f-327c-fce5-f457-202ee6e79a78}]"
2024-05-07T18:35:12.269Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77b15ebb-d061-c4f7-3c85-fd1f807a618a Address:77b15ebb-d061-c4f7-3c85-fd1f807a618a} {Suffrage:Voter ID:server-d22a04c1-dba3-4034-54ec-1bec7f3c84b9 Address:d22a04c1-dba3-4034-54ec-1bec7f3c84b9} {Suffrage:Voter ID:server-940b2c4f-327c-fce5-f457-202ee6e79a78 Address:940b2c4f-327c-fce5-f457-202ee6e79a78}]"
2024-05-07T18:35:12.269Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:12.269Z [INFO]  server-d22a04c1-dba3-4034-54ec-1bec7f3c84b9: entering follower state: follower="Node at d22a04c1-dba3-4034-54ec-1bec7f3c84b9 [Follower]" leader-address= leader-id=
2024-05-07T18:35:12.269Z [INFO]  server-77b15ebb-d061-c4f7-3c85-fd1f807a618a: entering follower state: follower="Node at 77b15ebb-d061-c4f7-3c85-fd1f807a618a [Follower]" leader-address= leader-id=
2024-05-07T18:35:12.269Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: entering follower state: follower="Node at 940b2c4f-327c-fce5-f457-202ee6e79a78 [Follower]" leader-address= leader-id=
2024-05-07T18:35:12.803Z [WARN]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:12.803Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: entering candidate state: node="Node at 940b2c4f-327c-fce5-f457-202ee6e79a78 [Candidate]" term=2
2024-05-07T18:35:12.803Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: election won: term=2 tally=2
2024-05-07T18:35:12.803Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: entering leader state: leader="Node at 940b2c4f-327c-fce5-f457-202ee6e79a78 [Leader]"
2024-05-07T18:35:12.803Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: added peer, starting replication: peer=server-77b15ebb-d061-c4f7-3c85-fd1f807a618a
2024-05-07T18:35:12.803Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: added peer, starting replication: peer=server-d22a04c1-dba3-4034-54ec-1bec7f3c84b9
2024-05-07T18:35:12.804Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: pipelining replication: peer="{Voter server-77b15ebb-d061-c4f7-3c85-fd1f807a618a 77b15ebb-d061-c4f7-3c85-fd1f807a618a}"
2024-05-07T18:35:12.804Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: pipelining replication: peer="{Voter server-d22a04c1-dba3-4034-54ec-1bec7f3c84b9 d22a04c1-dba3-4034-54ec-1bec7f3c84b9}"
2024-05-07T18:35:13.809Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:12.804666362 +0000 UTC m=+50.405784613 (1 nodes), 534.97813ms from start of poll, 535.259394ms from cluster start. Timeout at 2024-05-07 18:35:13.809928896 +0000 UTC m=+51.411047146, 1.005262533s after stability
2024-05-07T18:35:13.810Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: starting snapshot up to: index=12
2024-05-07T18:35:13.810Z [INFO]  creating new snapshot: path=/tmp/raft2829861504/snapshots/2-12-1715106913810.tmp
2024-05-07T18:35:13.848Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: snapshot complete up to: index=12
2024-05-07T18:35:13.848Z [INFO]  server-2e1ad299-ca58-b10d-6fe9-c7246439d566: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e1ad299-ca58-b10d-6fe9-c7246439d566 Address:2e1ad299-ca58-b10d-6fe9-c7246439d566} {Suffrage:Voter ID:server-aade5a4b-b9fa-9290-c420-801f981371dd Address:aade5a4b-b9fa-9290-c420-801f981371dd} {Suffrage:Voter ID:server-3277b719-c14b-a97c-7aa9-9f97c639d7dd Address:3277b719-c14b-a97c-7aa9-9f97c639d7dd}]"
2024-05-07T18:35:13.848Z [INFO]  server-aade5a4b-b9fa-9290-c420-801f981371dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e1ad299-ca58-b10d-6fe9-c7246439d566 Address:2e1ad299-ca58-b10d-6fe9-c7246439d566} {Suffrage:Voter ID:server-aade5a4b-b9fa-9290-c420-801f981371dd Address:aade5a4b-b9fa-9290-c420-801f981371dd} {Suffrage:Voter ID:server-3277b719-c14b-a97c-7aa9-9f97c639d7dd Address:3277b719-c14b-a97c-7aa9-9f97c639d7dd}]"
2024-05-07T18:35:13.848Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e1ad299-ca58-b10d-6fe9-c7246439d566 Address:2e1ad299-ca58-b10d-6fe9-c7246439d566} {Suffrage:Voter ID:server-aade5a4b-b9fa-9290-c420-801f981371dd Address:aade5a4b-b9fa-9290-c420-801f981371dd} {Suffrage:Voter ID:server-3277b719-c14b-a97c-7aa9-9f97c639d7dd Address:3277b719-c14b-a97c-7aa9-9f97c639d7dd}]"
2024-05-07T18:35:13.848Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:13.848Z [INFO]  server-2e1ad299-ca58-b10d-6fe9-c7246439d566: entering follower state: follower="Node at 2e1ad299-ca58-b10d-6fe9-c7246439d566 [Follower]" leader-address= leader-id=
2024-05-07T18:35:13.849Z [INFO]  server-aade5a4b-b9fa-9290-c420-801f981371dd: entering follower state: follower="Node at aade5a4b-b9fa-9290-c420-801f981371dd [Follower]" leader-address= leader-id=
2024-05-07T18:35:13.849Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: entering follower state: follower="Node at 3277b719-c14b-a97c-7aa9-9f97c639d7dd [Follower]" leader-address= leader-id=
2024-05-07T18:35:14.395Z [WARN]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:14.395Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: entering candidate state: node="Node at 3277b719-c14b-a97c-7aa9-9f97c639d7dd [Candidate]" term=2
2024-05-07T18:35:14.395Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: election won: term=2 tally=2
2024-05-07T18:35:14.395Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: entering leader state: leader="Node at 3277b719-c14b-a97c-7aa9-9f97c639d7dd [Leader]"
2024-05-07T18:35:14.395Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: added peer, starting replication: peer=server-2e1ad299-ca58-b10d-6fe9-c7246439d566
2024-05-07T18:35:14.395Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: added peer, starting replication: peer=server-aade5a4b-b9fa-9290-c420-801f981371dd
2024-05-07T18:35:14.395Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: pipelining replication: peer="{Voter server-aade5a4b-b9fa-9290-c420-801f981371dd aade5a4b-b9fa-9290-c420-801f981371dd}"
2024-05-07T18:35:14.395Z [WARN]  server-2e1ad299-ca58-b10d-6fe9-c7246439d566: rejecting vote request since our last term is greater: candidate=3277b719-c14b-a97c-7aa9-9f97c639d7dd last-term=2 last-candidate-term=1
2024-05-07T18:35:14.395Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: pipelining replication: peer="{Voter server-2e1ad299-ca58-b10d-6fe9-c7246439d566 2e1ad299-ca58-b10d-6fe9-c7246439d566}"
2024-05-07T18:35:15.401Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:14.395970742 +0000 UTC m=+51.997088992 (1 nodes), 546.99445ms from start of poll, 547.232082ms from cluster start. Timeout at 2024-05-07 18:35:15.401677461 +0000 UTC m=+53.002795701, 1.005706709s after stability
2024-05-07T18:35:15.418Z [INFO]  creating new snapshot: path=/tmp/raft234288453/snapshots/2-15-1715106915418.tmp
2024-05-07T18:35:15.566Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: copied to local snapshot: bytes=71
2024-05-07T18:35:15.566Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: snapshot restore progress: id=2-15-1715106915418 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:15.566Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: restored user snapshot: index=15
2024-05-07T18:35:15.566Z [ERROR] server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: failed to get log: index=3 error="log not found"
2024-05-07T18:35:15.566Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: aborting pipeline replication: peer="{Voter server-aade5a4b-b9fa-9290-c420-801f981371dd aade5a4b-b9fa-9290-c420-801f981371dd}"
2024-05-07T18:35:15.566Z [ERROR] server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: failed to get log: index=3 error="log not found"
2024-05-07T18:35:15.566Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: aborting pipeline replication: peer="{Voter server-2e1ad299-ca58-b10d-6fe9-c7246439d566 2e1ad299-ca58-b10d-6fe9-c7246439d566}"
2024-05-07T18:35:15.573Z [ERROR] server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: failed to get log: index=3 error="log not found"
2024-05-07T18:35:15.574Z [INFO]  creating new snapshot: path=/tmp/raft4107668123/snapshots/2-15-1715106915574.tmp
2024-05-07T18:35:15.574Z [INFO]  server-aade5a4b-b9fa-9290-c420-801f981371dd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:15.575Z [ERROR] server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: failed to get log: index=3 error="log not found"
2024-05-07T18:35:15.575Z [INFO]  creating new snapshot: path=/tmp/raft3699997092/snapshots/2-15-1715106915575.tmp
2024-05-07T18:35:15.575Z [INFO]  server-2e1ad299-ca58-b10d-6fe9-c7246439d566: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:15.624Z [INFO]  server-aade5a4b-b9fa-9290-c420-801f981371dd: copied to local snapshot: bytes=71
2024-05-07T18:35:15.624Z [INFO]  server-aade5a4b-b9fa-9290-c420-801f981371dd: snapshot restore progress: id=2-15-1715106915574 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:15.624Z [INFO]  server-aade5a4b-b9fa-9290-c420-801f981371dd: Installed remote snapshot
2024-05-07T18:35:15.624Z [INFO]  server-2e1ad299-ca58-b10d-6fe9-c7246439d566: copied to local snapshot: bytes=71
2024-05-07T18:35:15.624Z [INFO]  server-2e1ad299-ca58-b10d-6fe9-c7246439d566: snapshot restore progress: id=2-15-1715106915575 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:15.624Z [INFO]  server-2e1ad299-ca58-b10d-6fe9-c7246439d566: Installed remote snapshot
2024-05-07T18:35:15.624Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: pipelining replication: peer="{Voter server-aade5a4b-b9fa-9290-c420-801f981371dd aade5a4b-b9fa-9290-c420-801f981371dd}"
2024-05-07T18:35:15.625Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:15.625Z [INFO]  server-3277b719-c14b-a97c-7aa9-9f97c639d7dd: pipelining replication: peer="{Voter server-2e1ad299-ca58-b10d-6fe9-c7246439d566 2e1ad299-ca58-b10d-6fe9-c7246439d566}"
2024-05-07T18:35:16.630Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:15.625026558 +0000 UTC m=+53.226144808 (2 nodes), 982ns from start of poll, 1.776287898s from cluster start. Timeout at 2024-05-07 18:35:16.630039166 +0000 UTC m=+54.231157416, 1.005012608s after stability
2024-05-07T18:35:16.642Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: aborting pipeline replication: peer="{Voter server-77b15ebb-d061-c4f7-3c85-fd1f807a618a 77b15ebb-d061-c4f7-3c85-fd1f807a618a}"
2024-05-07T18:35:16.642Z [INFO]  server-940b2c4f-327c-fce5-f457-202ee6e79a78: aborting pipeline replication: peer="{Voter server-d22a04c1-dba3-4034-54ec-1bec7f3c84b9 d22a04c1-dba3-4034-54ec-1bec7f3c84b9}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2024-05-07T18:35:16.644Z [INFO]  server-cfcb9ee4-7192-da1c-bdd2-8a3d7a824735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfcb9ee4-7192-da1c-bdd2-8a3d7a824735 Address:cfcb9ee4-7192-da1c-bdd2-8a3d7a824735} {Suffrage:Voter ID:server-1d6ace30-7783-b736-5ed1-f46a76ce9610 Address:1d6ace30-7783-b736-5ed1-f46a76ce9610} {Suffrage:Voter ID:server-ac197c07-2fd4-ea62-79b4-77771a30f81f Address:ac197c07-2fd4-ea62-79b4-77771a30f81f}]"
2024-05-07T18:35:16.644Z [INFO]  server-cfcb9ee4-7192-da1c-bdd2-8a3d7a824735: entering follower state: follower="Node at cfcb9ee4-7192-da1c-bdd2-8a3d7a824735 [Follower]" leader-address= leader-id=
2024-05-07T18:35:16.644Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfcb9ee4-7192-da1c-bdd2-8a3d7a824735 Address:cfcb9ee4-7192-da1c-bdd2-8a3d7a824735} {Suffrage:Voter ID:server-1d6ace30-7783-b736-5ed1-f46a76ce9610 Address:1d6ace30-7783-b736-5ed1-f46a76ce9610} {Suffrage:Voter ID:server-ac197c07-2fd4-ea62-79b4-77771a30f81f Address:ac197c07-2fd4-ea62-79b4-77771a30f81f}]"
2024-05-07T18:35:16.644Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: entering follower state: follower="Node at 1d6ace30-7783-b736-5ed1-f46a76ce9610 [Follower]" leader-address= leader-id=
2024-05-07T18:35:16.644Z [INFO]  server-ac197c07-2fd4-ea62-79b4-77771a30f81f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfcb9ee4-7192-da1c-bdd2-8a3d7a824735 Address:cfcb9ee4-7192-da1c-bdd2-8a3d7a824735} {Suffrage:Voter ID:server-1d6ace30-7783-b736-5ed1-f46a76ce9610 Address:1d6ace30-7783-b736-5ed1-f46a76ce9610} {Suffrage:Voter ID:server-ac197c07-2fd4-ea62-79b4-77771a30f81f Address:ac197c07-2fd4-ea62-79b4-77771a30f81f}]"
2024-05-07T18:35:16.645Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:16.645Z [INFO]  server-ac197c07-2fd4-ea62-79b4-77771a30f81f: entering follower state: follower="Node at ac197c07-2fd4-ea62-79b4-77771a30f81f [Follower]" leader-address= leader-id=
2024-05-07T18:35:17.319Z [WARN]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:17.319Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: entering candidate state: node="Node at 1d6ace30-7783-b736-5ed1-f46a76ce9610 [Candidate]" term=2
2024-05-07T18:35:17.319Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: election won: term=2 tally=2
2024-05-07T18:35:17.319Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: entering leader state: leader="Node at 1d6ace30-7783-b736-5ed1-f46a76ce9610 [Leader]"
2024-05-07T18:35:17.319Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: added peer, starting replication: peer=server-cfcb9ee4-7192-da1c-bdd2-8a3d7a824735
2024-05-07T18:35:17.319Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: added peer, starting replication: peer=server-ac197c07-2fd4-ea62-79b4-77771a30f81f
2024-05-07T18:35:17.320Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: pipelining replication: peer="{Voter server-ac197c07-2fd4-ea62-79b4-77771a30f81f ac197c07-2fd4-ea62-79b4-77771a30f81f}"
2024-05-07T18:35:17.320Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: pipelining replication: peer="{Voter server-cfcb9ee4-7192-da1c-bdd2-8a3d7a824735 cfcb9ee4-7192-da1c-bdd2-8a3d7a824735}"
2024-05-07T18:35:18.325Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:17.320276389 +0000 UTC m=+54.921394639 (1 nodes), 675.233997ms from start of poll, 676.42328ms from cluster start. Timeout at 2024-05-07 18:35:18.325510631 +0000 UTC m=+55.926628881, 1.005234242s after stability
2024-05-07T18:35:18.325Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: starting snapshot up to: index=12
2024-05-07T18:35:18.325Z [INFO]  creating new snapshot: path=/tmp/raft3904963002/snapshots/2-12-1715106918325.tmp
2024-05-07T18:35:18.399Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: snapshot complete up to: index=12
2024-05-07T18:35:18.400Z [INFO]  server-cdf2694f-7dda-a64f-37dc-9bea4125a00f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdf2694f-7dda-a64f-37dc-9bea4125a00f Address:cdf2694f-7dda-a64f-37dc-9bea4125a00f} {Suffrage:Voter ID:server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2 Address:06d77ad1-0bcb-8022-7b8a-58fae90ab3a2} {Suffrage:Voter ID:server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4 Address:a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4}]"
2024-05-07T18:35:18.400Z [INFO]  server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdf2694f-7dda-a64f-37dc-9bea4125a00f Address:cdf2694f-7dda-a64f-37dc-9bea4125a00f} {Suffrage:Voter ID:server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2 Address:06d77ad1-0bcb-8022-7b8a-58fae90ab3a2} {Suffrage:Voter ID:server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4 Address:a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4}]"
2024-05-07T18:35:18.400Z [INFO]  server-cdf2694f-7dda-a64f-37dc-9bea4125a00f: entering follower state: follower="Node at cdf2694f-7dda-a64f-37dc-9bea4125a00f [Follower]" leader-address= leader-id=
2024-05-07T18:35:18.400Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdf2694f-7dda-a64f-37dc-9bea4125a00f Address:cdf2694f-7dda-a64f-37dc-9bea4125a00f} {Suffrage:Voter ID:server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2 Address:06d77ad1-0bcb-8022-7b8a-58fae90ab3a2} {Suffrage:Voter ID:server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4 Address:a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4}]"
2024-05-07T18:35:18.400Z [INFO]  server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2: entering follower state: follower="Node at 06d77ad1-0bcb-8022-7b8a-58fae90ab3a2 [Follower]" leader-address= leader-id=
2024-05-07T18:35:18.400Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:18.400Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: entering follower state: follower="Node at a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4 [Follower]" leader-address= leader-id=
2024-05-07T18:35:18.910Z [WARN]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:18.910Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: entering candidate state: node="Node at a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4 [Candidate]" term=2
2024-05-07T18:35:18.910Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: election won: term=2 tally=2
2024-05-07T18:35:18.910Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: entering leader state: leader="Node at a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4 [Leader]"
2024-05-07T18:35:18.910Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: added peer, starting replication: peer=server-cdf2694f-7dda-a64f-37dc-9bea4125a00f
2024-05-07T18:35:18.910Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: added peer, starting replication: peer=server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2
2024-05-07T18:35:18.910Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: pipelining replication: peer="{Voter server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2 06d77ad1-0bcb-8022-7b8a-58fae90ab3a2}"
2024-05-07T18:35:18.910Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: pipelining replication: peer="{Voter server-cdf2694f-7dda-a64f-37dc-9bea4125a00f cdf2694f-7dda-a64f-37dc-9bea4125a00f}"
2024-05-07T18:35:19.917Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:18.91081307 +0000 UTC m=+56.511931320 (1 nodes), 509.925482ms from start of poll, 510.229559ms from cluster start. Timeout at 2024-05-07 18:35:19.917253103 +0000 UTC m=+57.518371353, 1.006440033s after stability
2024-05-07T18:35:19.917Z [INFO]  creating new snapshot: path=/tmp/raft3547205080/snapshots/2-15-1715106919917.tmp
2024-05-07T18:35:19.917Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: copied to local snapshot: bytes=71
2024-05-07T18:35:19.918Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: snapshot restore progress: id=2-15-1715106919917 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:19.918Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: removing all old logs from log store
2024-05-07T18:35:19.918Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: compacting logs: from=1 to=2
2024-05-07T18:35:19.918Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: restored user snapshot: index=15
2024-05-07T18:35:19.918Z [ERROR] server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: failed to get log: index=2 error="log not found"
2024-05-07T18:35:19.918Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: aborting pipeline replication: peer="{Voter server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2 06d77ad1-0bcb-8022-7b8a-58fae90ab3a2}"
2024-05-07T18:35:19.918Z [ERROR] server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: failed to get log: index=2 error="log not found"
2024-05-07T18:35:19.918Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: aborting pipeline replication: peer="{Voter server-cdf2694f-7dda-a64f-37dc-9bea4125a00f cdf2694f-7dda-a64f-37dc-9bea4125a00f}"
2024-05-07T18:35:19.926Z [ERROR] server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: failed to get log: index=2 error="log not found"
2024-05-07T18:35:19.927Z [ERROR] server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: failed to get log: index=2 error="log not found"
2024-05-07T18:35:19.939Z [INFO]  creating new snapshot: path=/tmp/raft3716472791/snapshots/2-15-1715106919939.tmp
2024-05-07T18:35:19.939Z [INFO]  creating new snapshot: path=/tmp/raft3476119175/snapshots/2-15-1715106919939.tmp
2024-05-07T18:35:19.940Z [INFO]  server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:19.940Z [INFO]  server-cdf2694f-7dda-a64f-37dc-9bea4125a00f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:19.940Z [INFO]  server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2: copied to local snapshot: bytes=71
2024-05-07T18:35:19.940Z [INFO]  server-cdf2694f-7dda-a64f-37dc-9bea4125a00f: copied to local snapshot: bytes=71
2024-05-07T18:35:19.940Z [INFO]  server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2: snapshot restore progress: id=2-15-1715106919939 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:19.940Z [INFO]  server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2: removing all old logs from log store
2024-05-07T18:35:19.940Z [INFO]  server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2: compacting logs: from=1 to=2
2024-05-07T18:35:19.940Z [INFO]  server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2: Installed remote snapshot
2024-05-07T18:35:19.940Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: pipelining replication: peer="{Voter server-06d77ad1-0bcb-8022-7b8a-58fae90ab3a2 06d77ad1-0bcb-8022-7b8a-58fae90ab3a2}"
2024-05-07T18:35:19.940Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:19.940Z [INFO]  server-cdf2694f-7dda-a64f-37dc-9bea4125a00f: snapshot restore progress: id=2-15-1715106919939 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:19.940Z [INFO]  server-cdf2694f-7dda-a64f-37dc-9bea4125a00f: removing all old logs from log store
2024-05-07T18:35:19.940Z [INFO]  server-cdf2694f-7dda-a64f-37dc-9bea4125a00f: compacting logs: from=1 to=2
2024-05-07T18:35:19.940Z [INFO]  server-cdf2694f-7dda-a64f-37dc-9bea4125a00f: Installed remote snapshot
2024-05-07T18:35:19.941Z [INFO]  server-a0aa3e5e-c370-2af3-4bb3-2a4c352a5cd4: pipelining replication: peer="{Voter server-cdf2694f-7dda-a64f-37dc-9bea4125a00f cdf2694f-7dda-a64f-37dc-9bea4125a00f}"
2024-05-07T18:35:20.946Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:19.940847709 +0000 UTC m=+57.541965969 (2 nodes), 1.212µs from start of poll, 1.540264208s from cluster start. Timeout at 2024-05-07 18:35:20.946112004 +0000 UTC m=+58.547230254, 1.005264285s after stability
2024-05-07T18:35:20.958Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: aborting pipeline replication: peer="{Voter server-ac197c07-2fd4-ea62-79b4-77771a30f81f ac197c07-2fd4-ea62-79b4-77771a30f81f}"
2024-05-07T18:35:20.958Z [INFO]  server-1d6ace30-7783-b736-5ed1-f46a76ce9610: aborting pipeline replication: peer="{Voter server-cfcb9ee4-7192-da1c-bdd2-8a3d7a824735 cfcb9ee4-7192-da1c-bdd2-8a3d7a824735}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2024-05-07T18:35:20.966Z [INFO]  server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91 Address:4b430c9f-ef4c-7d62-b0ec-d44f60963c91} {Suffrage:Voter ID:server-788cc619-2789-c24f-fe35-8d95cb149b64 Address:788cc619-2789-c24f-fe35-8d95cb149b64} {Suffrage:Voter ID:server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f Address:c77bcebb-4bc5-9f53-ef2a-e986ed7be79f}]"
2024-05-07T18:35:20.966Z [INFO]  server-788cc619-2789-c24f-fe35-8d95cb149b64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91 Address:4b430c9f-ef4c-7d62-b0ec-d44f60963c91} {Suffrage:Voter ID:server-788cc619-2789-c24f-fe35-8d95cb149b64 Address:788cc619-2789-c24f-fe35-8d95cb149b64} {Suffrage:Voter ID:server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f Address:c77bcebb-4bc5-9f53-ef2a-e986ed7be79f}]"
2024-05-07T18:35:20.966Z [INFO]  server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91: entering follower state: follower="Node at 4b430c9f-ef4c-7d62-b0ec-d44f60963c91 [Follower]" leader-address= leader-id=
2024-05-07T18:35:20.966Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91 Address:4b430c9f-ef4c-7d62-b0ec-d44f60963c91} {Suffrage:Voter ID:server-788cc619-2789-c24f-fe35-8d95cb149b64 Address:788cc619-2789-c24f-fe35-8d95cb149b64} {Suffrage:Voter ID:server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f Address:c77bcebb-4bc5-9f53-ef2a-e986ed7be79f}]"
2024-05-07T18:35:20.966Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:20.966Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: entering follower state: follower="Node at c77bcebb-4bc5-9f53-ef2a-e986ed7be79f [Follower]" leader-address= leader-id=
2024-05-07T18:35:20.966Z [INFO]  server-788cc619-2789-c24f-fe35-8d95cb149b64: entering follower state: follower="Node at 788cc619-2789-c24f-fe35-8d95cb149b64 [Follower]" leader-address= leader-id=
2024-05-07T18:35:21.747Z [WARN]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:21.747Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: entering candidate state: node="Node at c77bcebb-4bc5-9f53-ef2a-e986ed7be79f [Candidate]" term=2
2024-05-07T18:35:21.747Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: election won: term=2 tally=2
2024-05-07T18:35:21.747Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: entering leader state: leader="Node at c77bcebb-4bc5-9f53-ef2a-e986ed7be79f [Leader]"
2024-05-07T18:35:21.747Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: added peer, starting replication: peer=server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91
2024-05-07T18:35:21.747Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: added peer, starting replication: peer=server-788cc619-2789-c24f-fe35-8d95cb149b64
2024-05-07T18:35:21.747Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: pipelining replication: peer="{Voter server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91 4b430c9f-ef4c-7d62-b0ec-d44f60963c91}"
2024-05-07T18:35:21.747Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: pipelining replication: peer="{Voter server-788cc619-2789-c24f-fe35-8d95cb149b64 788cc619-2789-c24f-fe35-8d95cb149b64}"
2024-05-07T18:35:22.752Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:21.747468704 +0000 UTC m=+59.348586964 (1 nodes), 780.849978ms from start of poll, 781.206381ms from cluster start. Timeout at 2024-05-07 18:35:22.752490139 +0000 UTC m=+60.353608389, 1.005021425s after stability
2024-05-07T18:35:22.753Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: starting snapshot up to: index=12
2024-05-07T18:35:22.753Z [INFO]  creating new snapshot: path=/tmp/raft3242318482/snapshots/2-12-1715106922753.tmp
2024-05-07T18:35:22.828Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: snapshot complete up to: index=12
2024-05-07T18:35:22.829Z [INFO]  creating new snapshot: path=/tmp/raft3242318482/snapshots/2-113-1715106922829.tmp
2024-05-07T18:35:22.829Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: copied to local snapshot: bytes=71
2024-05-07T18:35:22.830Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: snapshot restore progress: id=2-113-1715106922829 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:22.830Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: restored user snapshot: index=113
2024-05-07T18:35:22.830Z [ERROR] server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: failed to get log: index=23 error="log not found"
2024-05-07T18:35:22.830Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: aborting pipeline replication: peer="{Voter server-788cc619-2789-c24f-fe35-8d95cb149b64 788cc619-2789-c24f-fe35-8d95cb149b64}"
2024-05-07T18:35:22.830Z [ERROR] server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: failed to get log: index=23 error="log not found"
2024-05-07T18:35:22.830Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: aborting pipeline replication: peer="{Voter server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91 4b430c9f-ef4c-7d62-b0ec-d44f60963c91}"
2024-05-07T18:35:22.836Z [ERROR] server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: failed to get log: index=23 error="log not found"
2024-05-07T18:35:22.836Z [ERROR] server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: failed to get log: index=23 error="log not found"
2024-05-07T18:35:22.837Z [INFO]  creating new snapshot: path=/tmp/raft3978174936/snapshots/2-113-1715106922837.tmp
2024-05-07T18:35:22.838Z [INFO]  server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:22.838Z [INFO]  server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91: copied to local snapshot: bytes=71
2024-05-07T18:35:22.838Z [INFO]  creating new snapshot: path=/tmp/raft4186349347/snapshots/2-113-1715106922838.tmp
2024-05-07T18:35:22.839Z [INFO]  server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91: snapshot restore progress: id=2-113-1715106922837 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:22.839Z [INFO]  server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91: Installed remote snapshot
2024-05-07T18:35:22.839Z [INFO]  server-788cc619-2789-c24f-fe35-8d95cb149b64: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:22.839Z [WARN]  server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91: skipping application of old log: index=22
2024-05-07T18:35:22.839Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: pipelining replication: peer="{Voter server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91 4b430c9f-ef4c-7d62-b0ec-d44f60963c91}"
2024-05-07T18:35:22.839Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:22.839Z [INFO]  server-788cc619-2789-c24f-fe35-8d95cb149b64: copied to local snapshot: bytes=71
2024-05-07T18:35:22.839Z [INFO]  server-788cc619-2789-c24f-fe35-8d95cb149b64: snapshot restore progress: id=2-113-1715106922838 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:22.839Z [INFO]  server-788cc619-2789-c24f-fe35-8d95cb149b64: Installed remote snapshot
2024-05-07T18:35:22.839Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: pipelining replication: peer="{Voter server-788cc619-2789-c24f-fe35-8d95cb149b64 788cc619-2789-c24f-fe35-8d95cb149b64}"
2024-05-07T18:35:23.845Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:22.839655551 +0000 UTC m=+60.440773812 (2 nodes), 1.393µs from start of poll, 1.873393229s from cluster start. Timeout at 2024-05-07 18:35:23.84541562 +0000 UTC m=+61.446533870, 1.005760058s after stability
2024-05-07T18:35:23.857Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: aborting pipeline replication: peer="{Voter server-788cc619-2789-c24f-fe35-8d95cb149b64 788cc619-2789-c24f-fe35-8d95cb149b64}"
2024-05-07T18:35:23.857Z [INFO]  server-c77bcebb-4bc5-9f53-ef2a-e986ed7be79f: aborting pipeline replication: peer="{Voter server-4b430c9f-ef4c-7d62-b0ec-d44f60963c91 4b430c9f-ef4c-7d62-b0ec-d44f60963c91}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2024-05-07T18:35:23.858Z [INFO]  server-d46fceed-8172-4a71-4401-0d8189853010: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d46fceed-8172-4a71-4401-0d8189853010 Address:d46fceed-8172-4a71-4401-0d8189853010} {Suffrage:Voter ID:server-0f968b3a-103a-6ede-4ae3-c7c320130ddf Address:0f968b3a-103a-6ede-4ae3-c7c320130ddf} {Suffrage:Voter ID:server-7106ea0e-c51b-fb0f-9273-a835c820f354 Address:7106ea0e-c51b-fb0f-9273-a835c820f354}]"
2024-05-07T18:35:23.859Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d46fceed-8172-4a71-4401-0d8189853010 Address:d46fceed-8172-4a71-4401-0d8189853010} {Suffrage:Voter ID:server-0f968b3a-103a-6ede-4ae3-c7c320130ddf Address:0f968b3a-103a-6ede-4ae3-c7c320130ddf} {Suffrage:Voter ID:server-7106ea0e-c51b-fb0f-9273-a835c820f354 Address:7106ea0e-c51b-fb0f-9273-a835c820f354}]"
2024-05-07T18:35:23.859Z [INFO]  server-d46fceed-8172-4a71-4401-0d8189853010: entering follower state: follower="Node at d46fceed-8172-4a71-4401-0d8189853010 [Follower]" leader-address= leader-id=
2024-05-07T18:35:23.859Z [INFO]  server-7106ea0e-c51b-fb0f-9273-a835c820f354: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d46fceed-8172-4a71-4401-0d8189853010 Address:d46fceed-8172-4a71-4401-0d8189853010} {Suffrage:Voter ID:server-0f968b3a-103a-6ede-4ae3-c7c320130ddf Address:0f968b3a-103a-6ede-4ae3-c7c320130ddf} {Suffrage:Voter ID:server-7106ea0e-c51b-fb0f-9273-a835c820f354 Address:7106ea0e-c51b-fb0f-9273-a835c820f354}]"
2024-05-07T18:35:23.859Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:23.859Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: entering follower state: follower="Node at 0f968b3a-103a-6ede-4ae3-c7c320130ddf [Follower]" leader-address= leader-id=
2024-05-07T18:35:23.859Z [INFO]  server-7106ea0e-c51b-fb0f-9273-a835c820f354: entering follower state: follower="Node at 7106ea0e-c51b-fb0f-9273-a835c820f354 [Follower]" leader-address= leader-id=
2024-05-07T18:35:24.483Z [WARN]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:24.483Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: entering candidate state: node="Node at 0f968b3a-103a-6ede-4ae3-c7c320130ddf [Candidate]" term=2
2024-05-07T18:35:24.483Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: election won: term=2 tally=2
2024-05-07T18:35:24.483Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: entering leader state: leader="Node at 0f968b3a-103a-6ede-4ae3-c7c320130ddf [Leader]"
2024-05-07T18:35:24.483Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: added peer, starting replication: peer=server-d46fceed-8172-4a71-4401-0d8189853010
2024-05-07T18:35:24.483Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: added peer, starting replication: peer=server-7106ea0e-c51b-fb0f-9273-a835c820f354
2024-05-07T18:35:24.483Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: pipelining replication: peer="{Voter server-7106ea0e-c51b-fb0f-9273-a835c820f354 7106ea0e-c51b-fb0f-9273-a835c820f354}"
2024-05-07T18:35:24.483Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: pipelining replication: peer="{Voter server-d46fceed-8172-4a71-4401-0d8189853010 d46fceed-8172-4a71-4401-0d8189853010}"
2024-05-07T18:35:25.488Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:24.483501915 +0000 UTC m=+62.084620165 (1 nodes), 624.294233ms from start of poll, 624.637341ms from cluster start. Timeout at 2024-05-07 18:35:25.488703504 +0000 UTC m=+63.089821754, 1.005201589s after stability
2024-05-07T18:35:25.492Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: starting snapshot up to: index=12
2024-05-07T18:35:25.492Z [INFO]  creating new snapshot: path=/tmp/raft619807257/snapshots/2-12-1715106925492.tmp
2024-05-07T18:35:25.533Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: snapshot complete up to: index=12
2024-05-07T18:35:25.534Z [INFO]  creating new snapshot: path=/tmp/raft619807257/snapshots/2-113-1715106925534.tmp
2024-05-07T18:35:25.544Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: copied to local snapshot: bytes=71
2024-05-07T18:35:25.545Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: snapshot restore progress: id=2-113-1715106925534 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:25.545Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: removing all old logs from log store
2024-05-07T18:35:25.545Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: compacting logs: from=1 to=22
2024-05-07T18:35:25.545Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: restored user snapshot: index=113
2024-05-07T18:35:25.545Z [ERROR] server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: failed to get log: index=22 error="log not found"
2024-05-07T18:35:25.545Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: aborting pipeline replication: peer="{Voter server-7106ea0e-c51b-fb0f-9273-a835c820f354 7106ea0e-c51b-fb0f-9273-a835c820f354}"
2024-05-07T18:35:25.545Z [ERROR] server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: failed to get log: index=22 error="log not found"
2024-05-07T18:35:25.545Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: aborting pipeline replication: peer="{Voter server-d46fceed-8172-4a71-4401-0d8189853010 d46fceed-8172-4a71-4401-0d8189853010}"
2024-05-07T18:35:25.551Z [ERROR] server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: failed to get log: index=22 error="log not found"
2024-05-07T18:35:25.552Z [ERROR] server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: failed to get log: index=22 error="log not found"
2024-05-07T18:35:25.552Z [INFO]  creating new snapshot: path=/tmp/raft3971617768/snapshots/2-113-1715106925552.tmp
2024-05-07T18:35:25.552Z [INFO]  server-7106ea0e-c51b-fb0f-9273-a835c820f354: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:25.552Z [INFO]  creating new snapshot: path=/tmp/raft2133922795/snapshots/2-113-1715106925552.tmp
2024-05-07T18:35:25.553Z [INFO]  server-d46fceed-8172-4a71-4401-0d8189853010: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:25.592Z [INFO]  server-7106ea0e-c51b-fb0f-9273-a835c820f354: copied to local snapshot: bytes=71
2024-05-07T18:35:25.592Z [INFO]  server-7106ea0e-c51b-fb0f-9273-a835c820f354: snapshot restore progress: id=2-113-1715106925552 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:25.592Z [INFO]  server-7106ea0e-c51b-fb0f-9273-a835c820f354: removing all old logs from log store
2024-05-07T18:35:25.592Z [INFO]  server-7106ea0e-c51b-fb0f-9273-a835c820f354: compacting logs: from=1 to=22
2024-05-07T18:35:25.592Z [INFO]  server-7106ea0e-c51b-fb0f-9273-a835c820f354: Installed remote snapshot
2024-05-07T18:35:25.593Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: pipelining replication: peer="{Voter server-7106ea0e-c51b-fb0f-9273-a835c820f354 7106ea0e-c51b-fb0f-9273-a835c820f354}"
2024-05-07T18:35:25.593Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:25.593Z [INFO]  server-d46fceed-8172-4a71-4401-0d8189853010: copied to local snapshot: bytes=71
2024-05-07T18:35:25.593Z [INFO]  server-d46fceed-8172-4a71-4401-0d8189853010: snapshot restore progress: id=2-113-1715106925552 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:25.593Z [INFO]  server-d46fceed-8172-4a71-4401-0d8189853010: removing all old logs from log store
2024-05-07T18:35:25.593Z [INFO]  server-d46fceed-8172-4a71-4401-0d8189853010: compacting logs: from=1 to=22
2024-05-07T18:35:25.593Z [INFO]  server-d46fceed-8172-4a71-4401-0d8189853010: Installed remote snapshot
2024-05-07T18:35:25.593Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: pipelining replication: peer="{Voter server-d46fceed-8172-4a71-4401-0d8189853010 d46fceed-8172-4a71-4401-0d8189853010}"
2024-05-07T18:35:26.598Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:25.59319862 +0000 UTC m=+63.194316870 (2 nodes), 1.413µs from start of poll, 1.734334046s from cluster start. Timeout at 2024-05-07 18:35:26.598564505 +0000 UTC m=+64.199682755, 1.005365885s after stability
2024-05-07T18:35:26.615Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: aborting pipeline replication: peer="{Voter server-7106ea0e-c51b-fb0f-9273-a835c820f354 7106ea0e-c51b-fb0f-9273-a835c820f354}"
2024-05-07T18:35:26.615Z [INFO]  server-0f968b3a-103a-6ede-4ae3-c7c320130ddf: aborting pipeline replication: peer="{Voter server-d46fceed-8172-4a71-4401-0d8189853010 d46fceed-8172-4a71-4401-0d8189853010}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2024-05-07T18:35:26.616Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89309026-fdd1-2f5f-30ce-8aacc4741c73 Address:89309026-fdd1-2f5f-30ce-8aacc4741c73} {Suffrage:Voter ID:server-0d47afd8-e773-f914-aabb-c4108bc8326f Address:0d47afd8-e773-f914-aabb-c4108bc8326f} {Suffrage:Voter ID:server-192779a4-6c7d-645c-5642-b9c2ea54df3b Address:192779a4-6c7d-645c-5642-b9c2ea54df3b}]"
2024-05-07T18:35:26.616Z [INFO]  server-0d47afd8-e773-f914-aabb-c4108bc8326f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89309026-fdd1-2f5f-30ce-8aacc4741c73 Address:89309026-fdd1-2f5f-30ce-8aacc4741c73} {Suffrage:Voter ID:server-0d47afd8-e773-f914-aabb-c4108bc8326f Address:0d47afd8-e773-f914-aabb-c4108bc8326f} {Suffrage:Voter ID:server-192779a4-6c7d-645c-5642-b9c2ea54df3b Address:192779a4-6c7d-645c-5642-b9c2ea54df3b}]"
2024-05-07T18:35:26.616Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: entering follower state: follower="Node at 89309026-fdd1-2f5f-30ce-8aacc4741c73 [Follower]" leader-address= leader-id=
2024-05-07T18:35:26.616Z [INFO]  server-192779a4-6c7d-645c-5642-b9c2ea54df3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89309026-fdd1-2f5f-30ce-8aacc4741c73 Address:89309026-fdd1-2f5f-30ce-8aacc4741c73} {Suffrage:Voter ID:server-0d47afd8-e773-f914-aabb-c4108bc8326f Address:0d47afd8-e773-f914-aabb-c4108bc8326f} {Suffrage:Voter ID:server-192779a4-6c7d-645c-5642-b9c2ea54df3b Address:192779a4-6c7d-645c-5642-b9c2ea54df3b}]"
2024-05-07T18:35:26.616Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:26.616Z [INFO]  server-0d47afd8-e773-f914-aabb-c4108bc8326f: entering follower state: follower="Node at 0d47afd8-e773-f914-aabb-c4108bc8326f [Follower]" leader-address= leader-id=
2024-05-07T18:35:26.616Z [INFO]  server-192779a4-6c7d-645c-5642-b9c2ea54df3b: entering follower state: follower="Node at 192779a4-6c7d-645c-5642-b9c2ea54df3b [Follower]" leader-address= leader-id=
2024-05-07T18:35:27.404Z [WARN]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:27.404Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: entering candidate state: node="Node at 89309026-fdd1-2f5f-30ce-8aacc4741c73 [Candidate]" term=2
2024-05-07T18:35:27.404Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: election won: term=2 tally=2
2024-05-07T18:35:27.404Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: entering leader state: leader="Node at 89309026-fdd1-2f5f-30ce-8aacc4741c73 [Leader]"
2024-05-07T18:35:27.404Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: added peer, starting replication: peer=server-0d47afd8-e773-f914-aabb-c4108bc8326f
2024-05-07T18:35:27.404Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: added peer, starting replication: peer=server-192779a4-6c7d-645c-5642-b9c2ea54df3b
2024-05-07T18:35:27.404Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: pipelining replication: peer="{Voter server-192779a4-6c7d-645c-5642-b9c2ea54df3b 192779a4-6c7d-645c-5642-b9c2ea54df3b}"
2024-05-07T18:35:27.404Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: pipelining replication: peer="{Voter server-0d47afd8-e773-f914-aabb-c4108bc8326f 0d47afd8-e773-f914-aabb-c4108bc8326f}"
2024-05-07T18:35:28.410Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:27.404820095 +0000 UTC m=+65.005938346 (1 nodes), 788.080877ms from start of poll, 788.371768ms from cluster start. Timeout at 2024-05-07 18:35:28.410510585 +0000 UTC m=+66.011628835, 1.005690489s after stability
2024-05-07T18:35:28.410Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: starting snapshot up to: index=12
2024-05-07T18:35:28.411Z [INFO]  creating new snapshot: path=/tmp/raft496201111/snapshots/2-12-1715106928411.tmp
2024-05-07T18:35:28.411Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: snapshot complete up to: index=12
2024-05-07T18:35:28.413Z [INFO]  server-c1d49c78-e091-d5b5-e562-913537ba5eaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1d49c78-e091-d5b5-e562-913537ba5eaa Address:c1d49c78-e091-d5b5-e562-913537ba5eaa} {Suffrage:Voter ID:server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709 Address:fb6813e2-b2f6-221d-1b5b-2f8b7e414709} {Suffrage:Voter ID:server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e Address:2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e}]"
2024-05-07T18:35:28.413Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1d49c78-e091-d5b5-e562-913537ba5eaa Address:c1d49c78-e091-d5b5-e562-913537ba5eaa} {Suffrage:Voter ID:server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709 Address:fb6813e2-b2f6-221d-1b5b-2f8b7e414709} {Suffrage:Voter ID:server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e Address:2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e}]"
2024-05-07T18:35:28.413Z [INFO]  server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1d49c78-e091-d5b5-e562-913537ba5eaa Address:c1d49c78-e091-d5b5-e562-913537ba5eaa} {Suffrage:Voter ID:server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709 Address:fb6813e2-b2f6-221d-1b5b-2f8b7e414709} {Suffrage:Voter ID:server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e Address:2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e}]"
2024-05-07T18:35:28.413Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:28.413Z [INFO]  server-c1d49c78-e091-d5b5-e562-913537ba5eaa: entering follower state: follower="Node at c1d49c78-e091-d5b5-e562-913537ba5eaa [Follower]" leader-address= leader-id=
2024-05-07T18:35:28.413Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: entering follower state: follower="Node at fb6813e2-b2f6-221d-1b5b-2f8b7e414709 [Follower]" leader-address= leader-id=
2024-05-07T18:35:28.413Z [INFO]  server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e: entering follower state: follower="Node at 2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e [Follower]" leader-address= leader-id=
2024-05-07T18:35:28.981Z [WARN]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:28.981Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: entering candidate state: node="Node at fb6813e2-b2f6-221d-1b5b-2f8b7e414709 [Candidate]" term=2
2024-05-07T18:35:28.981Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: election won: term=2 tally=2
2024-05-07T18:35:28.981Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: entering leader state: leader="Node at fb6813e2-b2f6-221d-1b5b-2f8b7e414709 [Leader]"
2024-05-07T18:35:28.981Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: added peer, starting replication: peer=server-c1d49c78-e091-d5b5-e562-913537ba5eaa
2024-05-07T18:35:28.981Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: added peer, starting replication: peer=server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e
2024-05-07T18:35:28.981Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: pipelining replication: peer="{Voter server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e 2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e}"
2024-05-07T18:35:28.981Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: pipelining replication: peer="{Voter server-c1d49c78-e091-d5b5-e562-913537ba5eaa c1d49c78-e091-d5b5-e562-913537ba5eaa}"
2024-05-07T18:35:29.988Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:28.981982822 +0000 UTC m=+66.583101082 (1 nodes), 568.608201ms from start of poll, 569.000009ms from cluster start. Timeout at 2024-05-07 18:35:29.987915661 +0000 UTC m=+67.589033921, 1.005932839s after stability
2024-05-07T18:35:30.150Z [INFO]  creating new snapshot: path=/tmp/raft1723110762/snapshots/2-113-1715106930150.tmp
2024-05-07T18:35:30.456Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: copied to local snapshot: bytes=71
2024-05-07T18:35:30.457Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: snapshot restore progress: id=2-113-1715106930150 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:30.457Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: restored user snapshot: index=113
2024-05-07T18:35:30.457Z [ERROR] server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: failed to get log: index=3 error="log not found"
2024-05-07T18:35:30.457Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: aborting pipeline replication: peer="{Voter server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e 2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e}"
2024-05-07T18:35:30.457Z [ERROR] server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: failed to get log: index=3 error="log not found"
2024-05-07T18:35:30.457Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: aborting pipeline replication: peer="{Voter server-c1d49c78-e091-d5b5-e562-913537ba5eaa c1d49c78-e091-d5b5-e562-913537ba5eaa}"
2024-05-07T18:35:30.465Z [ERROR] server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: failed to get log: index=3 error="log not found"
2024-05-07T18:35:30.465Z [INFO]  creating new snapshot: path=/tmp/raft750617552/snapshots/2-113-1715106930465.tmp
2024-05-07T18:35:30.465Z [INFO]  server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:30.466Z [ERROR] server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: failed to get log: index=3 error="log not found"
2024-05-07T18:35:30.467Z [INFO]  creating new snapshot: path=/tmp/raft1336613894/snapshots/2-113-1715106930467.tmp
2024-05-07T18:35:30.467Z [INFO]  server-c1d49c78-e091-d5b5-e562-913537ba5eaa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:30.600Z [INFO]  server-c1d49c78-e091-d5b5-e562-913537ba5eaa: copied to local snapshot: bytes=71
2024-05-07T18:35:30.601Z [INFO]  server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e: copied to local snapshot: bytes=71
2024-05-07T18:35:30.601Z [INFO]  server-c1d49c78-e091-d5b5-e562-913537ba5eaa: snapshot restore progress: id=2-113-1715106930467 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:30.601Z [INFO]  server-c1d49c78-e091-d5b5-e562-913537ba5eaa: Installed remote snapshot
2024-05-07T18:35:30.601Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: pipelining replication: peer="{Voter server-c1d49c78-e091-d5b5-e562-913537ba5eaa c1d49c78-e091-d5b5-e562-913537ba5eaa}"
2024-05-07T18:35:30.601Z [INFO]  server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e: snapshot restore progress: id=2-113-1715106930465 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:30.601Z [INFO]  server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e: Installed remote snapshot
2024-05-07T18:35:30.601Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:30.601Z [INFO]  server-fb6813e2-b2f6-221d-1b5b-2f8b7e414709: pipelining replication: peer="{Voter server-2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e 2eb2510c-7c64-3a7a-4ef0-7428d61e3a6e}"
2024-05-07T18:35:31.607Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:30.601460771 +0000 UTC m=+68.202579432 (2 nodes), 3.637µs from start of poll, 2.188478359s from cluster start. Timeout at 2024-05-07 18:35:31.607873634 +0000 UTC m=+69.208991884, 1.006412452s after stability
2024-05-07T18:35:31.619Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: aborting pipeline replication: peer="{Voter server-192779a4-6c7d-645c-5642-b9c2ea54df3b 192779a4-6c7d-645c-5642-b9c2ea54df3b}"
2024-05-07T18:35:31.619Z [INFO]  server-89309026-fdd1-2f5f-30ce-8aacc4741c73: aborting pipeline replication: peer="{Voter server-0d47afd8-e773-f914-aabb-c4108bc8326f 0d47afd8-e773-f914-aabb-c4108bc8326f}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2024-05-07T18:35:31.623Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3 Address:3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3} {Suffrage:Voter ID:server-1998dcf8-9a60-7d66-5ae4-afc20b245f90 Address:1998dcf8-9a60-7d66-5ae4-afc20b245f90} {Suffrage:Voter ID:server-d5711ee3-03d1-5902-63d8-93dbdc03c903 Address:d5711ee3-03d1-5902-63d8-93dbdc03c903}]"
2024-05-07T18:35:31.623Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: entering follower state: follower="Node at 3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3 [Follower]" leader-address= leader-id=
2024-05-07T18:35:31.623Z [INFO]  server-1998dcf8-9a60-7d66-5ae4-afc20b245f90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3 Address:3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3} {Suffrage:Voter ID:server-1998dcf8-9a60-7d66-5ae4-afc20b245f90 Address:1998dcf8-9a60-7d66-5ae4-afc20b245f90} {Suffrage:Voter ID:server-d5711ee3-03d1-5902-63d8-93dbdc03c903 Address:d5711ee3-03d1-5902-63d8-93dbdc03c903}]"
2024-05-07T18:35:31.624Z [INFO]  server-d5711ee3-03d1-5902-63d8-93dbdc03c903: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3 Address:3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3} {Suffrage:Voter ID:server-1998dcf8-9a60-7d66-5ae4-afc20b245f90 Address:1998dcf8-9a60-7d66-5ae4-afc20b245f90} {Suffrage:Voter ID:server-d5711ee3-03d1-5902-63d8-93dbdc03c903 Address:d5711ee3-03d1-5902-63d8-93dbdc03c903}]"
2024-05-07T18:35:31.624Z [INFO]  server-1998dcf8-9a60-7d66-5ae4-afc20b245f90: entering follower state: follower="Node at 1998dcf8-9a60-7d66-5ae4-afc20b245f90 [Follower]" leader-address= leader-id=
2024-05-07T18:35:31.624Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:31.624Z [INFO]  server-d5711ee3-03d1-5902-63d8-93dbdc03c903: entering follower state: follower="Node at d5711ee3-03d1-5902-63d8-93dbdc03c903 [Follower]" leader-address= leader-id=
2024-05-07T18:35:32.346Z [WARN]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:32.346Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: entering candidate state: node="Node at 3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3 [Candidate]" term=2
2024-05-07T18:35:32.346Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: election won: term=2 tally=2
2024-05-07T18:35:32.346Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: entering leader state: leader="Node at 3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3 [Leader]"
2024-05-07T18:35:32.346Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: added peer, starting replication: peer=server-1998dcf8-9a60-7d66-5ae4-afc20b245f90
2024-05-07T18:35:32.346Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: added peer, starting replication: peer=server-d5711ee3-03d1-5902-63d8-93dbdc03c903
2024-05-07T18:35:32.346Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: pipelining replication: peer="{Voter server-d5711ee3-03d1-5902-63d8-93dbdc03c903 d5711ee3-03d1-5902-63d8-93dbdc03c903}"
2024-05-07T18:35:32.347Z [WARN]  server-1998dcf8-9a60-7d66-5ae4-afc20b245f90: rejecting vote request since our last term is greater: candidate=3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3 last-term=2 last-candidate-term=1
2024-05-07T18:35:32.347Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: pipelining replication: peer="{Voter server-1998dcf8-9a60-7d66-5ae4-afc20b245f90 1998dcf8-9a60-7d66-5ae4-afc20b245f90}"
2024-05-07T18:35:33.352Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:32.347088012 +0000 UTC m=+69.948206272 (1 nodes), 723.011652ms from start of poll, 723.505059ms from cluster start. Timeout at 2024-05-07 18:35:33.352573319 +0000 UTC m=+70.953691579, 1.005485307s after stability
2024-05-07T18:35:33.353Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: starting snapshot up to: index=12
2024-05-07T18:35:33.353Z [INFO]  creating new snapshot: path=/tmp/raft1014296556/snapshots/2-12-1715106933353.tmp
2024-05-07T18:35:33.378Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: snapshot complete up to: index=12
2024-05-07T18:35:33.379Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307 Address:ac9ed3fc-5b26-8b7e-bbb4-72144ec11307} {Suffrage:Voter ID:server-956a7501-0b81-440e-7eca-c83806f05aec Address:956a7501-0b81-440e-7eca-c83806f05aec} {Suffrage:Voter ID:server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75 Address:29bab2ad-14b0-b7bf-6d40-2d0bd7189e75}]"
2024-05-07T18:35:33.379Z [INFO]  server-956a7501-0b81-440e-7eca-c83806f05aec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307 Address:ac9ed3fc-5b26-8b7e-bbb4-72144ec11307} {Suffrage:Voter ID:server-956a7501-0b81-440e-7eca-c83806f05aec Address:956a7501-0b81-440e-7eca-c83806f05aec} {Suffrage:Voter ID:server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75 Address:29bab2ad-14b0-b7bf-6d40-2d0bd7189e75}]"
2024-05-07T18:35:33.379Z [INFO]  server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307 Address:ac9ed3fc-5b26-8b7e-bbb4-72144ec11307} {Suffrage:Voter ID:server-956a7501-0b81-440e-7eca-c83806f05aec Address:956a7501-0b81-440e-7eca-c83806f05aec} {Suffrage:Voter ID:server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75 Address:29bab2ad-14b0-b7bf-6d40-2d0bd7189e75}]"
2024-05-07T18:35:33.379Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:33.379Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: entering follower state: follower="Node at ac9ed3fc-5b26-8b7e-bbb4-72144ec11307 [Follower]" leader-address= leader-id=
2024-05-07T18:35:33.379Z [INFO]  server-956a7501-0b81-440e-7eca-c83806f05aec: entering follower state: follower="Node at 956a7501-0b81-440e-7eca-c83806f05aec [Follower]" leader-address= leader-id=
2024-05-07T18:35:33.379Z [INFO]  server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75: entering follower state: follower="Node at 29bab2ad-14b0-b7bf-6d40-2d0bd7189e75 [Follower]" leader-address= leader-id=
2024-05-07T18:35:33.928Z [WARN]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:33.928Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: entering candidate state: node="Node at ac9ed3fc-5b26-8b7e-bbb4-72144ec11307 [Candidate]" term=2
2024-05-07T18:35:33.928Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: election won: term=2 tally=2
2024-05-07T18:35:33.928Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: entering leader state: leader="Node at ac9ed3fc-5b26-8b7e-bbb4-72144ec11307 [Leader]"
2024-05-07T18:35:33.928Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: added peer, starting replication: peer=server-956a7501-0b81-440e-7eca-c83806f05aec
2024-05-07T18:35:33.928Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: added peer, starting replication: peer=server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75
2024-05-07T18:35:33.928Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: pipelining replication: peer="{Voter server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75 29bab2ad-14b0-b7bf-6d40-2d0bd7189e75}"
2024-05-07T18:35:33.928Z [WARN]  server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75: rejecting vote request since our last term is greater: candidate=ac9ed3fc-5b26-8b7e-bbb4-72144ec11307 last-term=2 last-candidate-term=1
2024-05-07T18:35:33.928Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: pipelining replication: peer="{Voter server-956a7501-0b81-440e-7eca-c83806f05aec 956a7501-0b81-440e-7eca-c83806f05aec}"
2024-05-07T18:35:34.934Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:33.928597362 +0000 UTC m=+71.529715612 (1 nodes), 548.907698ms from start of poll, 549.205592ms from cluster start. Timeout at 2024-05-07 18:35:34.934529279 +0000 UTC m=+72.535647540, 1.005931928s after stability
2024-05-07T18:35:34.934Z [INFO]  creating new snapshot: path=/tmp/raft4216941768/snapshots/2-113-1715106934934.tmp
2024-05-07T18:35:34.935Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: copied to local snapshot: bytes=71
2024-05-07T18:35:34.935Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: snapshot restore progress: id=2-113-1715106934934 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:34.935Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: removing all old logs from log store
2024-05-07T18:35:34.935Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: compacting logs: from=1 to=2
2024-05-07T18:35:34.935Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: restored user snapshot: index=113
2024-05-07T18:35:34.935Z [ERROR] server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: failed to get log: index=2 error="log not found"
2024-05-07T18:35:34.935Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: aborting pipeline replication: peer="{Voter server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75 29bab2ad-14b0-b7bf-6d40-2d0bd7189e75}"
2024-05-07T18:35:34.935Z [ERROR] server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: failed to get log: index=2 error="log not found"
2024-05-07T18:35:34.935Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: aborting pipeline replication: peer="{Voter server-956a7501-0b81-440e-7eca-c83806f05aec 956a7501-0b81-440e-7eca-c83806f05aec}"
2024-05-07T18:35:34.946Z [ERROR] server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: failed to get log: index=2 error="log not found"
2024-05-07T18:35:34.946Z [ERROR] server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: failed to get log: index=2 error="log not found"
2024-05-07T18:35:34.946Z [INFO]  creating new snapshot: path=/tmp/raft2994487719/snapshots/2-113-1715106934946.tmp
2024-05-07T18:35:34.947Z [INFO]  server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:34.947Z [INFO]  creating new snapshot: path=/tmp/raft3365219157/snapshots/2-113-1715106934947.tmp
2024-05-07T18:35:34.947Z [INFO]  server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75: copied to local snapshot: bytes=71
2024-05-07T18:35:34.947Z [INFO]  server-956a7501-0b81-440e-7eca-c83806f05aec: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:34.947Z [INFO]  server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75: snapshot restore progress: id=2-113-1715106934946 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:34.947Z [INFO]  server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75: removing all old logs from log store
2024-05-07T18:35:34.947Z [INFO]  server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75: compacting logs: from=1 to=2
2024-05-07T18:35:34.947Z [INFO]  server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75: Installed remote snapshot
2024-05-07T18:35:34.947Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: pipelining replication: peer="{Voter server-29bab2ad-14b0-b7bf-6d40-2d0bd7189e75 29bab2ad-14b0-b7bf-6d40-2d0bd7189e75}"
2024-05-07T18:35:34.947Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:34.947Z [INFO]  server-956a7501-0b81-440e-7eca-c83806f05aec: copied to local snapshot: bytes=71
2024-05-07T18:35:34.947Z [INFO]  server-956a7501-0b81-440e-7eca-c83806f05aec: snapshot restore progress: id=2-113-1715106934947 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:34.947Z [INFO]  server-956a7501-0b81-440e-7eca-c83806f05aec: removing all old logs from log store
2024-05-07T18:35:34.947Z [INFO]  server-956a7501-0b81-440e-7eca-c83806f05aec: compacting logs: from=1 to=2
2024-05-07T18:35:34.947Z [INFO]  server-956a7501-0b81-440e-7eca-c83806f05aec: Installed remote snapshot
2024-05-07T18:35:34.947Z [INFO]  server-ac9ed3fc-5b26-8b7e-bbb4-72144ec11307: pipelining replication: peer="{Voter server-956a7501-0b81-440e-7eca-c83806f05aec 956a7501-0b81-440e-7eca-c83806f05aec}"
2024-05-07T18:35:35.953Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:34.947620549 +0000 UTC m=+72.548738809 (2 nodes), 751ns from start of poll, 1.568228789s from cluster start. Timeout at 2024-05-07 18:35:35.953155889 +0000 UTC m=+73.554274139, 1.00553533s after stability
2024-05-07T18:35:35.965Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: aborting pipeline replication: peer="{Voter server-d5711ee3-03d1-5902-63d8-93dbdc03c903 d5711ee3-03d1-5902-63d8-93dbdc03c903}"
2024-05-07T18:35:35.965Z [INFO]  server-3c24eb81-4f0b-d8e0-f20d-36fd13ca5bb3: aborting pipeline replication: peer="{Voter server-1998dcf8-9a60-7d66-5ae4-afc20b245f90 1998dcf8-9a60-7d66-5ae4-afc20b245f90}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2024-05-07T18:35:35.966Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ff2361d-dc12-32e8-f829-bf64622e8498 Address:5ff2361d-dc12-32e8-f829-bf64622e8498} {Suffrage:Voter ID:server-16d6a461-7bfe-c9fe-3194-ac9dab49a814 Address:16d6a461-7bfe-c9fe-3194-ac9dab49a814} {Suffrage:Voter ID:server-35528b57-f28f-58ec-5312-febd2ea4132a Address:35528b57-f28f-58ec-5312-febd2ea4132a}]"
2024-05-07T18:35:35.966Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: entering follower state: follower="Node at 5ff2361d-dc12-32e8-f829-bf64622e8498 [Follower]" leader-address= leader-id=
2024-05-07T18:35:35.966Z [INFO]  server-16d6a461-7bfe-c9fe-3194-ac9dab49a814: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ff2361d-dc12-32e8-f829-bf64622e8498 Address:5ff2361d-dc12-32e8-f829-bf64622e8498} {Suffrage:Voter ID:server-16d6a461-7bfe-c9fe-3194-ac9dab49a814 Address:16d6a461-7bfe-c9fe-3194-ac9dab49a814} {Suffrage:Voter ID:server-35528b57-f28f-58ec-5312-febd2ea4132a Address:35528b57-f28f-58ec-5312-febd2ea4132a}]"
2024-05-07T18:35:35.966Z [INFO]  server-16d6a461-7bfe-c9fe-3194-ac9dab49a814: entering follower state: follower="Node at 16d6a461-7bfe-c9fe-3194-ac9dab49a814 [Follower]" leader-address= leader-id=
2024-05-07T18:35:35.966Z [INFO]  server-35528b57-f28f-58ec-5312-febd2ea4132a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ff2361d-dc12-32e8-f829-bf64622e8498 Address:5ff2361d-dc12-32e8-f829-bf64622e8498} {Suffrage:Voter ID:server-16d6a461-7bfe-c9fe-3194-ac9dab49a814 Address:16d6a461-7bfe-c9fe-3194-ac9dab49a814} {Suffrage:Voter ID:server-35528b57-f28f-58ec-5312-febd2ea4132a Address:35528b57-f28f-58ec-5312-febd2ea4132a}]"
2024-05-07T18:35:35.966Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:35.966Z [INFO]  server-35528b57-f28f-58ec-5312-febd2ea4132a: entering follower state: follower="Node at 35528b57-f28f-58ec-5312-febd2ea4132a [Follower]" leader-address= leader-id=
2024-05-07T18:35:36.561Z [WARN]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:36.561Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: entering candidate state: node="Node at 5ff2361d-dc12-32e8-f829-bf64622e8498 [Candidate]" term=2
2024-05-07T18:35:36.561Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: election won: term=2 tally=2
2024-05-07T18:35:36.561Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: entering leader state: leader="Node at 5ff2361d-dc12-32e8-f829-bf64622e8498 [Leader]"
2024-05-07T18:35:36.561Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: added peer, starting replication: peer=server-16d6a461-7bfe-c9fe-3194-ac9dab49a814
2024-05-07T18:35:36.561Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: added peer, starting replication: peer=server-35528b57-f28f-58ec-5312-febd2ea4132a
2024-05-07T18:35:36.561Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: pipelining replication: peer="{Voter server-35528b57-f28f-58ec-5312-febd2ea4132a 35528b57-f28f-58ec-5312-febd2ea4132a}"
2024-05-07T18:35:36.561Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: pipelining replication: peer="{Voter server-16d6a461-7bfe-c9fe-3194-ac9dab49a814 16d6a461-7bfe-c9fe-3194-ac9dab49a814}"
2024-05-07T18:35:37.566Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:36.561405189 +0000 UTC m=+74.162523439 (1 nodes), 594.651589ms from start of poll, 595.13563ms from cluster start. Timeout at 2024-05-07 18:35:37.566425211 +0000 UTC m=+75.167543461, 1.005020022s after stability
2024-05-07T18:35:37.566Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: starting snapshot up to: index=12
2024-05-07T18:35:37.566Z [INFO]  creating new snapshot: path=/tmp/raft1856976564/snapshots/2-12-1715106937566.tmp
2024-05-07T18:35:37.567Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: snapshot complete up to: index=12
2024-05-07T18:35:37.567Z [INFO]  creating new snapshot: path=/tmp/raft1856976564/snapshots/2-1013-1715106937567.tmp
2024-05-07T18:35:37.729Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: copied to local snapshot: bytes=71
2024-05-07T18:35:37.729Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: snapshot restore progress: id=2-1013-1715106937567 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:37.729Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: restored user snapshot: index=1013
2024-05-07T18:35:37.729Z [ERROR] server-5ff2361d-dc12-32e8-f829-bf64622e8498: failed to get log: index=23 error="log not found"
2024-05-07T18:35:37.729Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: aborting pipeline replication: peer="{Voter server-35528b57-f28f-58ec-5312-febd2ea4132a 35528b57-f28f-58ec-5312-febd2ea4132a}"
2024-05-07T18:35:37.729Z [ERROR] server-5ff2361d-dc12-32e8-f829-bf64622e8498: failed to get log: index=23 error="log not found"
2024-05-07T18:35:37.729Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: aborting pipeline replication: peer="{Voter server-16d6a461-7bfe-c9fe-3194-ac9dab49a814 16d6a461-7bfe-c9fe-3194-ac9dab49a814}"
2024-05-07T18:35:37.737Z [ERROR] server-5ff2361d-dc12-32e8-f829-bf64622e8498: failed to get log: index=23 error="log not found"
2024-05-07T18:35:37.737Z [INFO]  creating new snapshot: path=/tmp/raft1295462800/snapshots/2-1013-1715106937737.tmp
2024-05-07T18:35:37.737Z [INFO]  server-35528b57-f28f-58ec-5312-febd2ea4132a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:37.737Z [ERROR] server-5ff2361d-dc12-32e8-f829-bf64622e8498: failed to get log: index=23 error="log not found"
2024-05-07T18:35:37.738Z [INFO]  creating new snapshot: path=/tmp/raft3216127597/snapshots/2-1013-1715106937738.tmp
2024-05-07T18:35:37.738Z [INFO]  server-16d6a461-7bfe-c9fe-3194-ac9dab49a814: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:37.797Z [INFO]  server-16d6a461-7bfe-c9fe-3194-ac9dab49a814: copied to local snapshot: bytes=71
2024-05-07T18:35:37.797Z [INFO]  server-35528b57-f28f-58ec-5312-febd2ea4132a: copied to local snapshot: bytes=71
2024-05-07T18:35:37.797Z [INFO]  server-16d6a461-7bfe-c9fe-3194-ac9dab49a814: snapshot restore progress: id=2-1013-1715106937738 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:37.797Z [INFO]  server-16d6a461-7bfe-c9fe-3194-ac9dab49a814: Installed remote snapshot
2024-05-07T18:35:37.797Z [INFO]  server-35528b57-f28f-58ec-5312-febd2ea4132a: snapshot restore progress: id=2-1013-1715106937737 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:37.797Z [INFO]  server-35528b57-f28f-58ec-5312-febd2ea4132a: Installed remote snapshot
2024-05-07T18:35:37.797Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: pipelining replication: peer="{Voter server-16d6a461-7bfe-c9fe-3194-ac9dab49a814 16d6a461-7bfe-c9fe-3194-ac9dab49a814}"
2024-05-07T18:35:37.797Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:37.797Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: pipelining replication: peer="{Voter server-35528b57-f28f-58ec-5312-febd2ea4132a 35528b57-f28f-58ec-5312-febd2ea4132a}"
2024-05-07T18:35:38.803Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:37.79771661 +0000 UTC m=+75.398834850 (2 nodes), 812ns from start of poll, 1.831447041s from cluster start. Timeout at 2024-05-07 18:35:38.80307457 +0000 UTC m=+76.404192820, 1.00535797s after stability
2024-05-07T18:35:38.814Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: aborting pipeline replication: peer="{Voter server-35528b57-f28f-58ec-5312-febd2ea4132a 35528b57-f28f-58ec-5312-febd2ea4132a}"
2024-05-07T18:35:38.814Z [INFO]  server-5ff2361d-dc12-32e8-f829-bf64622e8498: aborting pipeline replication: peer="{Voter server-16d6a461-7bfe-c9fe-3194-ac9dab49a814 16d6a461-7bfe-c9fe-3194-ac9dab49a814}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2024-05-07T18:35:38.836Z [INFO]  server-531043b6-91d8-4055-45a2-4be695bfca8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-531043b6-91d8-4055-45a2-4be695bfca8a Address:531043b6-91d8-4055-45a2-4be695bfca8a} {Suffrage:Voter ID:server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4 Address:91e0f20e-92f5-f2b7-b145-5c7901de2ba4} {Suffrage:Voter ID:server-5fddec6b-3754-44fb-40d5-80eba96f1cfd Address:5fddec6b-3754-44fb-40d5-80eba96f1cfd}]"
2024-05-07T18:35:38.836Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-531043b6-91d8-4055-45a2-4be695bfca8a Address:531043b6-91d8-4055-45a2-4be695bfca8a} {Suffrage:Voter ID:server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4 Address:91e0f20e-92f5-f2b7-b145-5c7901de2ba4} {Suffrage:Voter ID:server-5fddec6b-3754-44fb-40d5-80eba96f1cfd Address:5fddec6b-3754-44fb-40d5-80eba96f1cfd}]"
2024-05-07T18:35:38.836Z [INFO]  server-5fddec6b-3754-44fb-40d5-80eba96f1cfd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-531043b6-91d8-4055-45a2-4be695bfca8a Address:531043b6-91d8-4055-45a2-4be695bfca8a} {Suffrage:Voter ID:server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4 Address:91e0f20e-92f5-f2b7-b145-5c7901de2ba4} {Suffrage:Voter ID:server-5fddec6b-3754-44fb-40d5-80eba96f1cfd Address:5fddec6b-3754-44fb-40d5-80eba96f1cfd}]"
2024-05-07T18:35:38.836Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:38.836Z [INFO]  server-531043b6-91d8-4055-45a2-4be695bfca8a: entering follower state: follower="Node at 531043b6-91d8-4055-45a2-4be695bfca8a [Follower]" leader-address= leader-id=
2024-05-07T18:35:38.836Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: entering follower state: follower="Node at 91e0f20e-92f5-f2b7-b145-5c7901de2ba4 [Follower]" leader-address= leader-id=
2024-05-07T18:35:38.836Z [INFO]  server-5fddec6b-3754-44fb-40d5-80eba96f1cfd: entering follower state: follower="Node at 5fddec6b-3754-44fb-40d5-80eba96f1cfd [Follower]" leader-address= leader-id=
2024-05-07T18:35:39.463Z [WARN]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:39.463Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: entering candidate state: node="Node at 91e0f20e-92f5-f2b7-b145-5c7901de2ba4 [Candidate]" term=2
2024-05-07T18:35:39.463Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: election won: term=2 tally=2
2024-05-07T18:35:39.463Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: entering leader state: leader="Node at 91e0f20e-92f5-f2b7-b145-5c7901de2ba4 [Leader]"
2024-05-07T18:35:39.463Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: added peer, starting replication: peer=server-531043b6-91d8-4055-45a2-4be695bfca8a
2024-05-07T18:35:39.463Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: added peer, starting replication: peer=server-5fddec6b-3754-44fb-40d5-80eba96f1cfd
2024-05-07T18:35:39.463Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: pipelining replication: peer="{Voter server-5fddec6b-3754-44fb-40d5-80eba96f1cfd 5fddec6b-3754-44fb-40d5-80eba96f1cfd}"
2024-05-07T18:35:39.463Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: pipelining replication: peer="{Voter server-531043b6-91d8-4055-45a2-4be695bfca8a 531043b6-91d8-4055-45a2-4be695bfca8a}"
2024-05-07T18:35:40.468Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:39.463382426 +0000 UTC m=+77.064500676 (1 nodes), 626.968617ms from start of poll, 627.29284ms from cluster start. Timeout at 2024-05-07 18:35:40.468745445 +0000 UTC m=+78.069863685, 1.005363009s after stability
2024-05-07T18:35:40.469Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: starting snapshot up to: index=12
2024-05-07T18:35:40.469Z [INFO]  creating new snapshot: path=/tmp/raft1096192934/snapshots/2-12-1715106940469.tmp
2024-05-07T18:35:40.587Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: snapshot complete up to: index=12
2024-05-07T18:35:40.587Z [INFO]  creating new snapshot: path=/tmp/raft1096192934/snapshots/2-1013-1715106940587.tmp
2024-05-07T18:35:40.588Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: copied to local snapshot: bytes=71
2024-05-07T18:35:40.588Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: snapshot restore progress: id=2-1013-1715106940587 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:40.588Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: removing all old logs from log store
2024-05-07T18:35:40.588Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: compacting logs: from=1 to=22
2024-05-07T18:35:40.588Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: restored user snapshot: index=1013
2024-05-07T18:35:40.588Z [ERROR] server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: failed to get log: index=22 error="log not found"
2024-05-07T18:35:40.588Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: aborting pipeline replication: peer="{Voter server-5fddec6b-3754-44fb-40d5-80eba96f1cfd 5fddec6b-3754-44fb-40d5-80eba96f1cfd}"
2024-05-07T18:35:40.588Z [ERROR] server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: failed to get log: index=22 error="log not found"
2024-05-07T18:35:40.588Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: aborting pipeline replication: peer="{Voter server-531043b6-91d8-4055-45a2-4be695bfca8a 531043b6-91d8-4055-45a2-4be695bfca8a}"
2024-05-07T18:35:40.597Z [ERROR] server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: failed to get log: index=22 error="log not found"
2024-05-07T18:35:40.597Z [INFO]  creating new snapshot: path=/tmp/raft1535004488/snapshots/2-1013-1715106940597.tmp
2024-05-07T18:35:40.597Z [INFO]  server-531043b6-91d8-4055-45a2-4be695bfca8a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:40.599Z [ERROR] server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: failed to get log: index=22 error="log not found"
2024-05-07T18:35:40.599Z [INFO]  creating new snapshot: path=/tmp/raft3134977793/snapshots/2-1013-1715106940599.tmp
2024-05-07T18:35:40.600Z [INFO]  server-5fddec6b-3754-44fb-40d5-80eba96f1cfd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:40.706Z [INFO]  server-531043b6-91d8-4055-45a2-4be695bfca8a: copied to local snapshot: bytes=71
2024-05-07T18:35:40.706Z [INFO]  server-5fddec6b-3754-44fb-40d5-80eba96f1cfd: copied to local snapshot: bytes=71
2024-05-07T18:35:40.706Z [INFO]  server-531043b6-91d8-4055-45a2-4be695bfca8a: snapshot restore progress: id=2-1013-1715106940597 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:40.706Z [INFO]  server-531043b6-91d8-4055-45a2-4be695bfca8a: removing all old logs from log store
2024-05-07T18:35:40.706Z [INFO]  server-531043b6-91d8-4055-45a2-4be695bfca8a: compacting logs: from=1 to=22
2024-05-07T18:35:40.706Z [INFO]  server-531043b6-91d8-4055-45a2-4be695bfca8a: Installed remote snapshot
2024-05-07T18:35:40.706Z [WARN]  server-531043b6-91d8-4055-45a2-4be695bfca8a: skipping application of old log: index=22
2024-05-07T18:35:40.706Z [INFO]  server-5fddec6b-3754-44fb-40d5-80eba96f1cfd: snapshot restore progress: id=2-1013-1715106940599 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:40.706Z [INFO]  server-5fddec6b-3754-44fb-40d5-80eba96f1cfd: removing all old logs from log store
2024-05-07T18:35:40.706Z [INFO]  server-5fddec6b-3754-44fb-40d5-80eba96f1cfd: compacting logs: from=1 to=22
2024-05-07T18:35:40.706Z [INFO]  server-5fddec6b-3754-44fb-40d5-80eba96f1cfd: Installed remote snapshot
2024-05-07T18:35:40.706Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: pipelining replication: peer="{Voter server-531043b6-91d8-4055-45a2-4be695bfca8a 531043b6-91d8-4055-45a2-4be695bfca8a}"
2024-05-07T18:35:40.706Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:40.706Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: pipelining replication: peer="{Voter server-5fddec6b-3754-44fb-40d5-80eba96f1cfd 5fddec6b-3754-44fb-40d5-80eba96f1cfd}"
2024-05-07T18:35:41.711Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:40.706612585 +0000 UTC m=+78.307730835 (2 nodes), 1.132µs from start of poll, 1.870522999s from cluster start. Timeout at 2024-05-07 18:35:41.711872864 +0000 UTC m=+79.312991114, 1.005260279s after stability
2024-05-07T18:35:41.723Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: aborting pipeline replication: peer="{Voter server-531043b6-91d8-4055-45a2-4be695bfca8a 531043b6-91d8-4055-45a2-4be695bfca8a}"
2024-05-07T18:35:41.723Z [INFO]  server-91e0f20e-92f5-f2b7-b145-5c7901de2ba4: aborting pipeline replication: peer="{Voter server-5fddec6b-3754-44fb-40d5-80eba96f1cfd 5fddec6b-3754-44fb-40d5-80eba96f1cfd}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2024-05-07T18:35:41.731Z [INFO]  server-eddb24e3-a286-64bb-d868-73f607be6cd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eddb24e3-a286-64bb-d868-73f607be6cd4 Address:eddb24e3-a286-64bb-d868-73f607be6cd4} {Suffrage:Voter ID:server-083e506f-8bb4-3a85-91e6-d3e498d8d083 Address:083e506f-8bb4-3a85-91e6-d3e498d8d083} {Suffrage:Voter ID:server-74d38d8b-46d9-5948-7a58-ab388386faa8 Address:74d38d8b-46d9-5948-7a58-ab388386faa8}]"
2024-05-07T18:35:41.731Z [INFO]  server-083e506f-8bb4-3a85-91e6-d3e498d8d083: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eddb24e3-a286-64bb-d868-73f607be6cd4 Address:eddb24e3-a286-64bb-d868-73f607be6cd4} {Suffrage:Voter ID:server-083e506f-8bb4-3a85-91e6-d3e498d8d083 Address:083e506f-8bb4-3a85-91e6-d3e498d8d083} {Suffrage:Voter ID:server-74d38d8b-46d9-5948-7a58-ab388386faa8 Address:74d38d8b-46d9-5948-7a58-ab388386faa8}]"
2024-05-07T18:35:41.731Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eddb24e3-a286-64bb-d868-73f607be6cd4 Address:eddb24e3-a286-64bb-d868-73f607be6cd4} {Suffrage:Voter ID:server-083e506f-8bb4-3a85-91e6-d3e498d8d083 Address:083e506f-8bb4-3a85-91e6-d3e498d8d083} {Suffrage:Voter ID:server-74d38d8b-46d9-5948-7a58-ab388386faa8 Address:74d38d8b-46d9-5948-7a58-ab388386faa8}]"
2024-05-07T18:35:41.731Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:41.731Z [INFO]  server-eddb24e3-a286-64bb-d868-73f607be6cd4: entering follower state: follower="Node at eddb24e3-a286-64bb-d868-73f607be6cd4 [Follower]" leader-address= leader-id=
2024-05-07T18:35:41.731Z [INFO]  server-083e506f-8bb4-3a85-91e6-d3e498d8d083: entering follower state: follower="Node at 083e506f-8bb4-3a85-91e6-d3e498d8d083 [Follower]" leader-address= leader-id=
2024-05-07T18:35:41.731Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: entering follower state: follower="Node at 74d38d8b-46d9-5948-7a58-ab388386faa8 [Follower]" leader-address= leader-id=
2024-05-07T18:35:42.411Z [WARN]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:42.411Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: entering candidate state: node="Node at 74d38d8b-46d9-5948-7a58-ab388386faa8 [Candidate]" term=2
2024-05-07T18:35:42.411Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: election won: term=2 tally=2
2024-05-07T18:35:42.411Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: entering leader state: leader="Node at 74d38d8b-46d9-5948-7a58-ab388386faa8 [Leader]"
2024-05-07T18:35:42.411Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: added peer, starting replication: peer=server-eddb24e3-a286-64bb-d868-73f607be6cd4
2024-05-07T18:35:42.411Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: added peer, starting replication: peer=server-083e506f-8bb4-3a85-91e6-d3e498d8d083
2024-05-07T18:35:42.411Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: pipelining replication: peer="{Voter server-083e506f-8bb4-3a85-91e6-d3e498d8d083 083e506f-8bb4-3a85-91e6-d3e498d8d083}"
2024-05-07T18:35:42.413Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: pipelining replication: peer="{Voter server-eddb24e3-a286-64bb-d868-73f607be6cd4 eddb24e3-a286-64bb-d868-73f607be6cd4}"
2024-05-07T18:35:43.418Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:42.413353559 +0000 UTC m=+80.014471809 (1 nodes), 681.779161ms from start of poll, 682.032764ms from cluster start. Timeout at 2024-05-07 18:35:43.418359534 +0000 UTC m=+81.019477784, 1.005005975s after stability
2024-05-07T18:35:43.418Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: starting snapshot up to: index=12
2024-05-07T18:35:43.418Z [INFO]  creating new snapshot: path=/tmp/raft3738537702/snapshots/2-12-1715106943418.tmp
2024-05-07T18:35:43.457Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: snapshot complete up to: index=12
2024-05-07T18:35:43.459Z [INFO]  server-def374a1-c177-7013-99cf-a75fe6b1cf6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-def374a1-c177-7013-99cf-a75fe6b1cf6d Address:def374a1-c177-7013-99cf-a75fe6b1cf6d} {Suffrage:Voter ID:server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29 Address:0efb79f7-b9b4-8c2a-360d-56ab90b93d29} {Suffrage:Voter ID:server-e0fe80c7-156f-e0f0-125c-b4e985d827fe Address:e0fe80c7-156f-e0f0-125c-b4e985d827fe}]"
2024-05-07T18:35:43.459Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-def374a1-c177-7013-99cf-a75fe6b1cf6d Address:def374a1-c177-7013-99cf-a75fe6b1cf6d} {Suffrage:Voter ID:server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29 Address:0efb79f7-b9b4-8c2a-360d-56ab90b93d29} {Suffrage:Voter ID:server-e0fe80c7-156f-e0f0-125c-b4e985d827fe Address:e0fe80c7-156f-e0f0-125c-b4e985d827fe}]"
2024-05-07T18:35:43.460Z [INFO]  server-e0fe80c7-156f-e0f0-125c-b4e985d827fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-def374a1-c177-7013-99cf-a75fe6b1cf6d Address:def374a1-c177-7013-99cf-a75fe6b1cf6d} {Suffrage:Voter ID:server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29 Address:0efb79f7-b9b4-8c2a-360d-56ab90b93d29} {Suffrage:Voter ID:server-e0fe80c7-156f-e0f0-125c-b4e985d827fe Address:e0fe80c7-156f-e0f0-125c-b4e985d827fe}]"
2024-05-07T18:35:43.460Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:43.462Z [INFO]  server-def374a1-c177-7013-99cf-a75fe6b1cf6d: entering follower state: follower="Node at def374a1-c177-7013-99cf-a75fe6b1cf6d [Follower]" leader-address= leader-id=
2024-05-07T18:35:43.462Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: entering follower state: follower="Node at 0efb79f7-b9b4-8c2a-360d-56ab90b93d29 [Follower]" leader-address= leader-id=
2024-05-07T18:35:43.462Z [INFO]  server-e0fe80c7-156f-e0f0-125c-b4e985d827fe: entering follower state: follower="Node at e0fe80c7-156f-e0f0-125c-b4e985d827fe [Follower]" leader-address= leader-id=
2024-05-07T18:35:44.230Z [WARN]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:44.230Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: entering candidate state: node="Node at 0efb79f7-b9b4-8c2a-360d-56ab90b93d29 [Candidate]" term=2
2024-05-07T18:35:44.230Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: election won: term=2 tally=2
2024-05-07T18:35:44.230Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: entering leader state: leader="Node at 0efb79f7-b9b4-8c2a-360d-56ab90b93d29 [Leader]"
2024-05-07T18:35:44.230Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: added peer, starting replication: peer=server-def374a1-c177-7013-99cf-a75fe6b1cf6d
2024-05-07T18:35:44.230Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: added peer, starting replication: peer=server-e0fe80c7-156f-e0f0-125c-b4e985d827fe
2024-05-07T18:35:44.230Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: pipelining replication: peer="{Voter server-e0fe80c7-156f-e0f0-125c-b4e985d827fe e0fe80c7-156f-e0f0-125c-b4e985d827fe}"
2024-05-07T18:35:44.230Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: pipelining replication: peer="{Voter server-def374a1-c177-7013-99cf-a75fe6b1cf6d def374a1-c177-7013-99cf-a75fe6b1cf6d}"
2024-05-07T18:35:45.235Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:44.230330425 +0000 UTC m=+81.831448675 (1 nodes), 770.225257ms from start of poll, 771.181665ms from cluster start. Timeout at 2024-05-07 18:35:45.235671734 +0000 UTC m=+82.836789984, 1.005341309s after stability
2024-05-07T18:35:45.236Z [INFO]  creating new snapshot: path=/tmp/raft2509423706/snapshots/2-1013-1715106945236.tmp
2024-05-07T18:35:45.315Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: copied to local snapshot: bytes=71
2024-05-07T18:35:45.315Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: snapshot restore progress: id=2-1013-1715106945236 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:45.315Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: restored user snapshot: index=1013
2024-05-07T18:35:45.315Z [ERROR] server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: failed to get log: index=3 error="log not found"
2024-05-07T18:35:45.315Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: aborting pipeline replication: peer="{Voter server-e0fe80c7-156f-e0f0-125c-b4e985d827fe e0fe80c7-156f-e0f0-125c-b4e985d827fe}"
2024-05-07T18:35:45.315Z [ERROR] server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: failed to get log: index=3 error="log not found"
2024-05-07T18:35:45.315Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: aborting pipeline replication: peer="{Voter server-def374a1-c177-7013-99cf-a75fe6b1cf6d def374a1-c177-7013-99cf-a75fe6b1cf6d}"
2024-05-07T18:35:45.324Z [ERROR] server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: failed to get log: index=3 error="log not found"
2024-05-07T18:35:45.325Z [INFO]  creating new snapshot: path=/tmp/raft217817196/snapshots/2-1013-1715106945325.tmp
2024-05-07T18:35:45.325Z [INFO]  server-e0fe80c7-156f-e0f0-125c-b4e985d827fe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:45.325Z [ERROR] server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: failed to get log: index=3 error="log not found"
2024-05-07T18:35:45.325Z [INFO]  creating new snapshot: path=/tmp/raft3690441835/snapshots/2-1013-1715106945325.tmp
2024-05-07T18:35:45.325Z [INFO]  server-def374a1-c177-7013-99cf-a75fe6b1cf6d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:45.419Z [INFO]  server-def374a1-c177-7013-99cf-a75fe6b1cf6d: copied to local snapshot: bytes=71
2024-05-07T18:35:45.420Z [INFO]  server-def374a1-c177-7013-99cf-a75fe6b1cf6d: snapshot restore progress: id=2-1013-1715106945325 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:45.420Z [INFO]  server-def374a1-c177-7013-99cf-a75fe6b1cf6d: Installed remote snapshot
2024-05-07T18:35:45.420Z [INFO]  server-e0fe80c7-156f-e0f0-125c-b4e985d827fe: copied to local snapshot: bytes=71
2024-05-07T18:35:45.420Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: pipelining replication: peer="{Voter server-def374a1-c177-7013-99cf-a75fe6b1cf6d def374a1-c177-7013-99cf-a75fe6b1cf6d}"
2024-05-07T18:35:45.420Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:45.420Z [INFO]  server-e0fe80c7-156f-e0f0-125c-b4e985d827fe: snapshot restore progress: id=2-1013-1715106945325 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:45.420Z [INFO]  server-e0fe80c7-156f-e0f0-125c-b4e985d827fe: Installed remote snapshot
2024-05-07T18:35:45.420Z [INFO]  server-0efb79f7-b9b4-8c2a-360d-56ab90b93d29: pipelining replication: peer="{Voter server-e0fe80c7-156f-e0f0-125c-b4e985d827fe e0fe80c7-156f-e0f0-125c-b4e985d827fe}"
2024-05-07T18:35:46.425Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:45.420240822 +0000 UTC m=+83.021359072 (2 nodes), 1.082µs from start of poll, 1.961092062s from cluster start. Timeout at 2024-05-07 18:35:46.425428236 +0000 UTC m=+84.026546486, 1.005187414s after stability
2024-05-07T18:35:46.437Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: aborting pipeline replication: peer="{Voter server-083e506f-8bb4-3a85-91e6-d3e498d8d083 083e506f-8bb4-3a85-91e6-d3e498d8d083}"
2024-05-07T18:35:46.437Z [INFO]  server-74d38d8b-46d9-5948-7a58-ab388386faa8: aborting pipeline replication: peer="{Voter server-eddb24e3-a286-64bb-d868-73f607be6cd4 eddb24e3-a286-64bb-d868-73f607be6cd4}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2024-05-07T18:35:46.467Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c90e488b-b8ad-6080-b01b-b04782496d30 Address:c90e488b-b8ad-6080-b01b-b04782496d30} {Suffrage:Voter ID:server-b12202f2-af51-cf7a-4063-10bc1a67b618 Address:b12202f2-af51-cf7a-4063-10bc1a67b618} {Suffrage:Voter ID:server-e35e0821-a152-6380-c479-3ac7f8a81d07 Address:e35e0821-a152-6380-c479-3ac7f8a81d07}]"
2024-05-07T18:35:46.467Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: entering follower state: follower="Node at c90e488b-b8ad-6080-b01b-b04782496d30 [Follower]" leader-address= leader-id=
2024-05-07T18:35:46.467Z [INFO]  server-b12202f2-af51-cf7a-4063-10bc1a67b618: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c90e488b-b8ad-6080-b01b-b04782496d30 Address:c90e488b-b8ad-6080-b01b-b04782496d30} {Suffrage:Voter ID:server-b12202f2-af51-cf7a-4063-10bc1a67b618 Address:b12202f2-af51-cf7a-4063-10bc1a67b618} {Suffrage:Voter ID:server-e35e0821-a152-6380-c479-3ac7f8a81d07 Address:e35e0821-a152-6380-c479-3ac7f8a81d07}]"
2024-05-07T18:35:46.468Z [INFO]  server-b12202f2-af51-cf7a-4063-10bc1a67b618: entering follower state: follower="Node at b12202f2-af51-cf7a-4063-10bc1a67b618 [Follower]" leader-address= leader-id=
2024-05-07T18:35:46.468Z [INFO]  server-e35e0821-a152-6380-c479-3ac7f8a81d07: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c90e488b-b8ad-6080-b01b-b04782496d30 Address:c90e488b-b8ad-6080-b01b-b04782496d30} {Suffrage:Voter ID:server-b12202f2-af51-cf7a-4063-10bc1a67b618 Address:b12202f2-af51-cf7a-4063-10bc1a67b618} {Suffrage:Voter ID:server-e35e0821-a152-6380-c479-3ac7f8a81d07 Address:e35e0821-a152-6380-c479-3ac7f8a81d07}]"
2024-05-07T18:35:46.468Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:46.468Z [INFO]  server-e35e0821-a152-6380-c479-3ac7f8a81d07: entering follower state: follower="Node at e35e0821-a152-6380-c479-3ac7f8a81d07 [Follower]" leader-address= leader-id=
2024-05-07T18:35:47.068Z [WARN]  server-c90e488b-b8ad-6080-b01b-b04782496d30: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:47.068Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: entering candidate state: node="Node at c90e488b-b8ad-6080-b01b-b04782496d30 [Candidate]" term=2
2024-05-07T18:35:47.068Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: election won: term=2 tally=2
2024-05-07T18:35:47.068Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: entering leader state: leader="Node at c90e488b-b8ad-6080-b01b-b04782496d30 [Leader]"
2024-05-07T18:35:47.068Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: added peer, starting replication: peer=server-b12202f2-af51-cf7a-4063-10bc1a67b618
2024-05-07T18:35:47.068Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: added peer, starting replication: peer=server-e35e0821-a152-6380-c479-3ac7f8a81d07
2024-05-07T18:35:47.068Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: pipelining replication: peer="{Voter server-e35e0821-a152-6380-c479-3ac7f8a81d07 e35e0821-a152-6380-c479-3ac7f8a81d07}"
2024-05-07T18:35:47.068Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: pipelining replication: peer="{Voter server-b12202f2-af51-cf7a-4063-10bc1a67b618 b12202f2-af51-cf7a-4063-10bc1a67b618}"
2024-05-07T18:35:48.074Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:47.068849703 +0000 UTC m=+84.669967953 (1 nodes), 600.633815ms from start of poll, 601.133504ms from cluster start. Timeout at 2024-05-07 18:35:48.074319992 +0000 UTC m=+85.675438252, 1.005470299s after stability
2024-05-07T18:35:48.075Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: starting snapshot up to: index=12
2024-05-07T18:35:48.075Z [INFO]  creating new snapshot: path=/tmp/raft2902272263/snapshots/2-12-1715106948075.tmp
2024-05-07T18:35:48.140Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: snapshot complete up to: index=12
2024-05-07T18:35:48.141Z [INFO]  server-86b15f64-d577-17fe-2b0b-75637a9b80bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86b15f64-d577-17fe-2b0b-75637a9b80bb Address:86b15f64-d577-17fe-2b0b-75637a9b80bb} {Suffrage:Voter ID:server-4350f676-e302-5dfa-5dbf-e58a3aac22d2 Address:4350f676-e302-5dfa-5dbf-e58a3aac22d2} {Suffrage:Voter ID:server-c95b2f88-ae99-4dde-8f50-a1665649db6a Address:c95b2f88-ae99-4dde-8f50-a1665649db6a}]"
2024-05-07T18:35:48.141Z [INFO]  server-4350f676-e302-5dfa-5dbf-e58a3aac22d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86b15f64-d577-17fe-2b0b-75637a9b80bb Address:86b15f64-d577-17fe-2b0b-75637a9b80bb} {Suffrage:Voter ID:server-4350f676-e302-5dfa-5dbf-e58a3aac22d2 Address:4350f676-e302-5dfa-5dbf-e58a3aac22d2} {Suffrage:Voter ID:server-c95b2f88-ae99-4dde-8f50-a1665649db6a Address:c95b2f88-ae99-4dde-8f50-a1665649db6a}]"
2024-05-07T18:35:48.141Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86b15f64-d577-17fe-2b0b-75637a9b80bb Address:86b15f64-d577-17fe-2b0b-75637a9b80bb} {Suffrage:Voter ID:server-4350f676-e302-5dfa-5dbf-e58a3aac22d2 Address:4350f676-e302-5dfa-5dbf-e58a3aac22d2} {Suffrage:Voter ID:server-c95b2f88-ae99-4dde-8f50-a1665649db6a Address:c95b2f88-ae99-4dde-8f50-a1665649db6a}]"
2024-05-07T18:35:48.141Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:48.141Z [INFO]  server-86b15f64-d577-17fe-2b0b-75637a9b80bb: entering follower state: follower="Node at 86b15f64-d577-17fe-2b0b-75637a9b80bb [Follower]" leader-address= leader-id=
2024-05-07T18:35:48.141Z [INFO]  server-4350f676-e302-5dfa-5dbf-e58a3aac22d2: entering follower state: follower="Node at 4350f676-e302-5dfa-5dbf-e58a3aac22d2 [Follower]" leader-address= leader-id=
2024-05-07T18:35:48.141Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: entering follower state: follower="Node at c95b2f88-ae99-4dde-8f50-a1665649db6a [Follower]" leader-address= leader-id=
2024-05-07T18:35:48.816Z [WARN]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:48.816Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: entering candidate state: node="Node at c95b2f88-ae99-4dde-8f50-a1665649db6a [Candidate]" term=2
2024-05-07T18:35:48.816Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: election won: term=2 tally=2
2024-05-07T18:35:48.816Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: entering leader state: leader="Node at c95b2f88-ae99-4dde-8f50-a1665649db6a [Leader]"
2024-05-07T18:35:48.816Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: added peer, starting replication: peer=server-86b15f64-d577-17fe-2b0b-75637a9b80bb
2024-05-07T18:35:48.816Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: added peer, starting replication: peer=server-4350f676-e302-5dfa-5dbf-e58a3aac22d2
2024-05-07T18:35:48.816Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: pipelining replication: peer="{Voter server-4350f676-e302-5dfa-5dbf-e58a3aac22d2 4350f676-e302-5dfa-5dbf-e58a3aac22d2}"
2024-05-07T18:35:48.816Z [WARN]  server-4350f676-e302-5dfa-5dbf-e58a3aac22d2: rejecting vote request since our last term is greater: candidate=c95b2f88-ae99-4dde-8f50-a1665649db6a last-term=2 last-candidate-term=1
2024-05-07T18:35:48.816Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: pipelining replication: peer="{Voter server-86b15f64-d577-17fe-2b0b-75637a9b80bb 86b15f64-d577-17fe-2b0b-75637a9b80bb}"
2024-05-07T18:35:49.821Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:48.816624778 +0000 UTC m=+86.417743028 (1 nodes), 674.799359ms from start of poll, 675.052991ms from cluster start. Timeout at 2024-05-07 18:35:49.821640341 +0000 UTC m=+87.422758591, 1.005015563s after stability
2024-05-07T18:35:49.821Z [INFO]  creating new snapshot: path=/tmp/raft865466360/snapshots/2-1013-1715106949821.tmp
2024-05-07T18:35:49.822Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: copied to local snapshot: bytes=71
2024-05-07T18:35:49.822Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: snapshot restore progress: id=2-1013-1715106949821 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:49.822Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: removing all old logs from log store
2024-05-07T18:35:49.822Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: compacting logs: from=1 to=2
2024-05-07T18:35:49.822Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: restored user snapshot: index=1013
2024-05-07T18:35:49.822Z [ERROR] server-c95b2f88-ae99-4dde-8f50-a1665649db6a: failed to get log: index=2 error="log not found"
2024-05-07T18:35:49.822Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: aborting pipeline replication: peer="{Voter server-4350f676-e302-5dfa-5dbf-e58a3aac22d2 4350f676-e302-5dfa-5dbf-e58a3aac22d2}"
2024-05-07T18:35:49.822Z [ERROR] server-c95b2f88-ae99-4dde-8f50-a1665649db6a: failed to get log: index=2 error="log not found"
2024-05-07T18:35:49.822Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: aborting pipeline replication: peer="{Voter server-86b15f64-d577-17fe-2b0b-75637a9b80bb 86b15f64-d577-17fe-2b0b-75637a9b80bb}"
2024-05-07T18:35:49.829Z [ERROR] server-c95b2f88-ae99-4dde-8f50-a1665649db6a: failed to get log: index=2 error="log not found"
2024-05-07T18:35:49.829Z [INFO]  creating new snapshot: path=/tmp/raft3916793771/snapshots/2-1013-1715106949829.tmp
2024-05-07T18:35:49.830Z [INFO]  server-86b15f64-d577-17fe-2b0b-75637a9b80bb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:49.831Z [ERROR] server-c95b2f88-ae99-4dde-8f50-a1665649db6a: failed to get log: index=2 error="log not found"
2024-05-07T18:35:49.832Z [INFO]  creating new snapshot: path=/tmp/raft1313293139/snapshots/2-1013-1715106949832.tmp
2024-05-07T18:35:49.832Z [INFO]  server-4350f676-e302-5dfa-5dbf-e58a3aac22d2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:49.919Z [INFO]  server-4350f676-e302-5dfa-5dbf-e58a3aac22d2: copied to local snapshot: bytes=71
2024-05-07T18:35:49.919Z [INFO]  server-86b15f64-d577-17fe-2b0b-75637a9b80bb: copied to local snapshot: bytes=71
2024-05-07T18:35:49.919Z [INFO]  server-4350f676-e302-5dfa-5dbf-e58a3aac22d2: snapshot restore progress: id=2-1013-1715106949832 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:49.919Z [INFO]  server-4350f676-e302-5dfa-5dbf-e58a3aac22d2: removing all old logs from log store
2024-05-07T18:35:49.919Z [INFO]  server-4350f676-e302-5dfa-5dbf-e58a3aac22d2: compacting logs: from=1 to=2
2024-05-07T18:35:49.919Z [INFO]  server-86b15f64-d577-17fe-2b0b-75637a9b80bb: snapshot restore progress: id=2-1013-1715106949829 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:49.919Z [INFO]  server-86b15f64-d577-17fe-2b0b-75637a9b80bb: removing all old logs from log store
2024-05-07T18:35:49.919Z [INFO]  server-86b15f64-d577-17fe-2b0b-75637a9b80bb: compacting logs: from=1 to=2
2024-05-07T18:35:49.919Z [INFO]  server-86b15f64-d577-17fe-2b0b-75637a9b80bb: Installed remote snapshot
2024-05-07T18:35:49.919Z [INFO]  server-4350f676-e302-5dfa-5dbf-e58a3aac22d2: Installed remote snapshot
2024-05-07T18:35:49.919Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: pipelining replication: peer="{Voter server-86b15f64-d577-17fe-2b0b-75637a9b80bb 86b15f64-d577-17fe-2b0b-75637a9b80bb}"
2024-05-07T18:35:49.919Z [INFO]  server-c95b2f88-ae99-4dde-8f50-a1665649db6a: pipelining replication: peer="{Voter server-4350f676-e302-5dfa-5dbf-e58a3aac22d2 4350f676-e302-5dfa-5dbf-e58a3aac22d2}"
2024-05-07T18:35:49.919Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:50.924Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:49.919568362 +0000 UTC m=+87.520686612 (2 nodes), 1.052µs from start of poll, 1.777996575s from cluster start. Timeout at 2024-05-07 18:35:50.924590226 +0000 UTC m=+88.525708476, 1.005021864s after stability
2024-05-07T18:35:50.936Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: aborting pipeline replication: peer="{Voter server-e35e0821-a152-6380-c479-3ac7f8a81d07 e35e0821-a152-6380-c479-3ac7f8a81d07}"
2024-05-07T18:35:50.936Z [INFO]  server-c90e488b-b8ad-6080-b01b-b04782496d30: aborting pipeline replication: peer="{Voter server-b12202f2-af51-cf7a-4063-10bc1a67b618 b12202f2-af51-cf7a-4063-10bc1a67b618}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2024-05-07T18:35:50.941Z [INFO]  server-c82e059e-6993-b878-b64d-f8b8f24fd818: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c82e059e-6993-b878-b64d-f8b8f24fd818 Address:c82e059e-6993-b878-b64d-f8b8f24fd818} {Suffrage:Voter ID:server-eb81a4cc-244d-d6bc-8021-7c020e8d8237 Address:eb81a4cc-244d-d6bc-8021-7c020e8d8237} {Suffrage:Voter ID:server-eb933bdb-5237-5116-c359-8d539d71b615 Address:eb933bdb-5237-5116-c359-8d539d71b615}]"
2024-05-07T18:35:50.941Z [INFO]  server-c82e059e-6993-b878-b64d-f8b8f24fd818: entering follower state: follower="Node at c82e059e-6993-b878-b64d-f8b8f24fd818 [Follower]" leader-address= leader-id=
2024-05-07T18:35:50.941Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c82e059e-6993-b878-b64d-f8b8f24fd818 Address:c82e059e-6993-b878-b64d-f8b8f24fd818} {Suffrage:Voter ID:server-eb81a4cc-244d-d6bc-8021-7c020e8d8237 Address:eb81a4cc-244d-d6bc-8021-7c020e8d8237} {Suffrage:Voter ID:server-eb933bdb-5237-5116-c359-8d539d71b615 Address:eb933bdb-5237-5116-c359-8d539d71b615}]"
2024-05-07T18:35:50.941Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: entering follower state: follower="Node at eb81a4cc-244d-d6bc-8021-7c020e8d8237 [Follower]" leader-address= leader-id=
2024-05-07T18:35:50.941Z [INFO]  server-eb933bdb-5237-5116-c359-8d539d71b615: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c82e059e-6993-b878-b64d-f8b8f24fd818 Address:c82e059e-6993-b878-b64d-f8b8f24fd818} {Suffrage:Voter ID:server-eb81a4cc-244d-d6bc-8021-7c020e8d8237 Address:eb81a4cc-244d-d6bc-8021-7c020e8d8237} {Suffrage:Voter ID:server-eb933bdb-5237-5116-c359-8d539d71b615 Address:eb933bdb-5237-5116-c359-8d539d71b615}]"
2024-05-07T18:35:50.941Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:50.941Z [INFO]  server-eb933bdb-5237-5116-c359-8d539d71b615: entering follower state: follower="Node at eb933bdb-5237-5116-c359-8d539d71b615 [Follower]" leader-address= leader-id=
2024-05-07T18:35:51.582Z [WARN]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:51.582Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: entering candidate state: node="Node at eb81a4cc-244d-d6bc-8021-7c020e8d8237 [Candidate]" term=2
2024-05-07T18:35:51.582Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: election won: term=2 tally=2
2024-05-07T18:35:51.582Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: entering leader state: leader="Node at eb81a4cc-244d-d6bc-8021-7c020e8d8237 [Leader]"
2024-05-07T18:35:51.582Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: added peer, starting replication: peer=server-c82e059e-6993-b878-b64d-f8b8f24fd818
2024-05-07T18:35:51.582Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: added peer, starting replication: peer=server-eb933bdb-5237-5116-c359-8d539d71b615
2024-05-07T18:35:51.582Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: pipelining replication: peer="{Voter server-eb933bdb-5237-5116-c359-8d539d71b615 eb933bdb-5237-5116-c359-8d539d71b615}"
2024-05-07T18:35:51.582Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: pipelining replication: peer="{Voter server-c82e059e-6993-b878-b64d-f8b8f24fd818 c82e059e-6993-b878-b64d-f8b8f24fd818}"
2024-05-07T18:35:52.588Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:51.582696528 +0000 UTC m=+89.183814788 (1 nodes), 641.181612ms from start of poll, 641.44891ms from cluster start. Timeout at 2024-05-07 18:35:52.588553646 +0000 UTC m=+90.189671896, 1.005857108s after stability
2024-05-07T18:35:52.588Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: starting snapshot up to: index=12
2024-05-07T18:35:52.588Z [INFO]  creating new snapshot: path=/tmp/raft2821671878/snapshots/2-12-1715106952588.tmp
2024-05-07T18:35:52.589Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: snapshot complete up to: index=12
2024-05-07T18:35:52.589Z [INFO]  creating new snapshot: path=/tmp/raft2821671878/snapshots/2-10013-1715106952589.tmp
2024-05-07T18:35:52.590Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: copied to local snapshot: bytes=71
2024-05-07T18:35:52.590Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: snapshot restore progress: id=2-10013-1715106952589 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:52.590Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: restored user snapshot: index=10013
2024-05-07T18:35:52.590Z [ERROR] server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: failed to get log: index=23 error="log not found"
2024-05-07T18:35:52.590Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: aborting pipeline replication: peer="{Voter server-eb933bdb-5237-5116-c359-8d539d71b615 eb933bdb-5237-5116-c359-8d539d71b615}"
2024-05-07T18:35:52.590Z [ERROR] server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: failed to get log: index=23 error="log not found"
2024-05-07T18:35:52.590Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: aborting pipeline replication: peer="{Voter server-c82e059e-6993-b878-b64d-f8b8f24fd818 c82e059e-6993-b878-b64d-f8b8f24fd818}"
2024-05-07T18:35:52.599Z [ERROR] server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: failed to get log: index=23 error="log not found"
2024-05-07T18:35:52.599Z [ERROR] server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: failed to get log: index=23 error="log not found"
2024-05-07T18:35:52.599Z [INFO]  creating new snapshot: path=/tmp/raft1978179465/snapshots/2-10013-1715106952599.tmp
2024-05-07T18:35:52.599Z [INFO]  server-eb933bdb-5237-5116-c359-8d539d71b615: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:52.599Z [INFO]  server-eb933bdb-5237-5116-c359-8d539d71b615: copied to local snapshot: bytes=71
2024-05-07T18:35:52.599Z [INFO]  creating new snapshot: path=/tmp/raft260384965/snapshots/2-10013-1715106952599.tmp
2024-05-07T18:35:52.600Z [INFO]  server-c82e059e-6993-b878-b64d-f8b8f24fd818: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:52.600Z [INFO]  server-c82e059e-6993-b878-b64d-f8b8f24fd818: copied to local snapshot: bytes=71
2024-05-07T18:35:52.600Z [INFO]  server-c82e059e-6993-b878-b64d-f8b8f24fd818: snapshot restore progress: id=2-10013-1715106952599 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:52.600Z [INFO]  server-c82e059e-6993-b878-b64d-f8b8f24fd818: Installed remote snapshot
2024-05-07T18:35:52.600Z [WARN]  server-c82e059e-6993-b878-b64d-f8b8f24fd818: skipping application of old log: index=22
2024-05-07T18:35:52.600Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: pipelining replication: peer="{Voter server-c82e059e-6993-b878-b64d-f8b8f24fd818 c82e059e-6993-b878-b64d-f8b8f24fd818}"
2024-05-07T18:35:52.600Z [INFO]  server-eb933bdb-5237-5116-c359-8d539d71b615: snapshot restore progress: id=2-10013-1715106952599 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:52.600Z [INFO]  server-eb933bdb-5237-5116-c359-8d539d71b615: Installed remote snapshot
2024-05-07T18:35:52.600Z [WARN]  server-eb933bdb-5237-5116-c359-8d539d71b615: skipping application of old log: index=22
2024-05-07T18:35:52.600Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: pipelining replication: peer="{Voter server-eb933bdb-5237-5116-c359-8d539d71b615 eb933bdb-5237-5116-c359-8d539d71b615}"
2024-05-07T18:35:52.600Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:53.606Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:52.600646118 +0000 UTC m=+90.201764368 (2 nodes), 431ns from start of poll, 1.65939849s from cluster start. Timeout at 2024-05-07 18:35:53.606175997 +0000 UTC m=+91.207294247, 1.005529879s after stability
2024-05-07T18:35:53.618Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: aborting pipeline replication: peer="{Voter server-eb933bdb-5237-5116-c359-8d539d71b615 eb933bdb-5237-5116-c359-8d539d71b615}"
2024-05-07T18:35:53.618Z [INFO]  server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: aborting pipeline replication: peer="{Voter server-c82e059e-6993-b878-b64d-f8b8f24fd818 c82e059e-6993-b878-b64d-f8b8f24fd818}"
2024-05-07T18:35:54.119Z [ERROR] server-eb81a4cc-244d-d6bc-8021-7c020e8d8237: failed to heartbeat to: peer=eb933bdb-5237-5116-c359-8d539d71b615 backoff time=10ms error="command timed out"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2024-05-07T18:35:54.122Z [INFO]  server-55663e76-e997-46fe-cd32-3e96399e67c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55663e76-e997-46fe-cd32-3e96399e67c5 Address:55663e76-e997-46fe-cd32-3e96399e67c5} {Suffrage:Voter ID:server-75893e81-3d86-37c0-55ba-3e9c36a0908b Address:75893e81-3d86-37c0-55ba-3e9c36a0908b} {Suffrage:Voter ID:server-2efc90e0-72f9-1a01-aa60-e7520589d4b3 Address:2efc90e0-72f9-1a01-aa60-e7520589d4b3}]"
2024-05-07T18:35:54.122Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55663e76-e997-46fe-cd32-3e96399e67c5 Address:55663e76-e997-46fe-cd32-3e96399e67c5} {Suffrage:Voter ID:server-75893e81-3d86-37c0-55ba-3e9c36a0908b Address:75893e81-3d86-37c0-55ba-3e9c36a0908b} {Suffrage:Voter ID:server-2efc90e0-72f9-1a01-aa60-e7520589d4b3 Address:2efc90e0-72f9-1a01-aa60-e7520589d4b3}]"
2024-05-07T18:35:54.122Z [INFO]  server-2efc90e0-72f9-1a01-aa60-e7520589d4b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55663e76-e997-46fe-cd32-3e96399e67c5 Address:55663e76-e997-46fe-cd32-3e96399e67c5} {Suffrage:Voter ID:server-75893e81-3d86-37c0-55ba-3e9c36a0908b Address:75893e81-3d86-37c0-55ba-3e9c36a0908b} {Suffrage:Voter ID:server-2efc90e0-72f9-1a01-aa60-e7520589d4b3 Address:2efc90e0-72f9-1a01-aa60-e7520589d4b3}]"
2024-05-07T18:35:54.122Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:54.122Z [INFO]  server-55663e76-e997-46fe-cd32-3e96399e67c5: entering follower state: follower="Node at 55663e76-e997-46fe-cd32-3e96399e67c5 [Follower]" leader-address= leader-id=
2024-05-07T18:35:54.123Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: entering follower state: follower="Node at 75893e81-3d86-37c0-55ba-3e9c36a0908b [Follower]" leader-address= leader-id=
2024-05-07T18:35:54.123Z [INFO]  server-2efc90e0-72f9-1a01-aa60-e7520589d4b3: entering follower state: follower="Node at 2efc90e0-72f9-1a01-aa60-e7520589d4b3 [Follower]" leader-address= leader-id=
2024-05-07T18:35:54.628Z [WARN]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:54.628Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: entering candidate state: node="Node at 75893e81-3d86-37c0-55ba-3e9c36a0908b [Candidate]" term=2
2024-05-07T18:35:54.630Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: election won: term=2 tally=2
2024-05-07T18:35:54.630Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: entering leader state: leader="Node at 75893e81-3d86-37c0-55ba-3e9c36a0908b [Leader]"
2024-05-07T18:35:54.630Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: added peer, starting replication: peer=server-55663e76-e997-46fe-cd32-3e96399e67c5
2024-05-07T18:35:54.630Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: added peer, starting replication: peer=server-2efc90e0-72f9-1a01-aa60-e7520589d4b3
2024-05-07T18:35:54.631Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: pipelining replication: peer="{Voter server-2efc90e0-72f9-1a01-aa60-e7520589d4b3 2efc90e0-72f9-1a01-aa60-e7520589d4b3}"
2024-05-07T18:35:54.631Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: pipelining replication: peer="{Voter server-55663e76-e997-46fe-cd32-3e96399e67c5 55663e76-e997-46fe-cd32-3e96399e67c5}"
2024-05-07T18:35:55.636Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:54.631146937 +0000 UTC m=+92.232265187 (1 nodes), 508.174527ms from start of poll, 508.430152ms from cluster start. Timeout at 2024-05-07 18:35:55.636807379 +0000 UTC m=+93.237925629, 1.005660442s after stability
2024-05-07T18:35:55.637Z [WARN]  server-55663e76-e997-46fe-cd32-3e96399e67c5: skipping application of old log: index=2
2024-05-07T18:35:55.637Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: starting snapshot up to: index=12
2024-05-07T18:35:55.637Z [INFO]  creating new snapshot: path=/tmp/raft1147000002/snapshots/2-12-1715106955637.tmp
2024-05-07T18:35:55.655Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: snapshot complete up to: index=12
2024-05-07T18:35:55.656Z [INFO]  creating new snapshot: path=/tmp/raft1147000002/snapshots/2-10013-1715106955656.tmp
2024-05-07T18:35:55.656Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: copied to local snapshot: bytes=71
2024-05-07T18:35:55.656Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: snapshot restore progress: id=2-10013-1715106955656 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:55.656Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: removing all old logs from log store
2024-05-07T18:35:55.656Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: compacting logs: from=1 to=22
2024-05-07T18:35:55.656Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: restored user snapshot: index=10013
2024-05-07T18:35:55.656Z [ERROR] server-75893e81-3d86-37c0-55ba-3e9c36a0908b: failed to get log: index=22 error="log not found"
2024-05-07T18:35:55.656Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: aborting pipeline replication: peer="{Voter server-2efc90e0-72f9-1a01-aa60-e7520589d4b3 2efc90e0-72f9-1a01-aa60-e7520589d4b3}"
2024-05-07T18:35:55.656Z [ERROR] server-75893e81-3d86-37c0-55ba-3e9c36a0908b: failed to get log: index=22 error="log not found"
2024-05-07T18:35:55.656Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: aborting pipeline replication: peer="{Voter server-55663e76-e997-46fe-cd32-3e96399e67c5 55663e76-e997-46fe-cd32-3e96399e67c5}"
2024-05-07T18:35:55.662Z [ERROR] server-75893e81-3d86-37c0-55ba-3e9c36a0908b: failed to get log: index=22 error="log not found"
2024-05-07T18:35:55.663Z [INFO]  creating new snapshot: path=/tmp/raft3480626714/snapshots/2-10013-1715106955663.tmp
2024-05-07T18:35:55.663Z [INFO]  server-55663e76-e997-46fe-cd32-3e96399e67c5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:55.667Z [ERROR] server-75893e81-3d86-37c0-55ba-3e9c36a0908b: failed to get log: index=22 error="log not found"
2024-05-07T18:35:55.667Z [INFO]  creating new snapshot: path=/tmp/raft2326388007/snapshots/2-10013-1715106955667.tmp
2024-05-07T18:35:55.667Z [INFO]  server-2efc90e0-72f9-1a01-aa60-e7520589d4b3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:55.758Z [INFO]  server-55663e76-e997-46fe-cd32-3e96399e67c5: copied to local snapshot: bytes=71
2024-05-07T18:35:55.758Z [INFO]  server-2efc90e0-72f9-1a01-aa60-e7520589d4b3: copied to local snapshot: bytes=71
2024-05-07T18:35:55.758Z [INFO]  server-55663e76-e997-46fe-cd32-3e96399e67c5: snapshot restore progress: id=2-10013-1715106955663 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:55.758Z [INFO]  server-55663e76-e997-46fe-cd32-3e96399e67c5: removing all old logs from log store
2024-05-07T18:35:55.758Z [INFO]  server-55663e76-e997-46fe-cd32-3e96399e67c5: compacting logs: from=1 to=22
2024-05-07T18:35:55.759Z [INFO]  server-55663e76-e997-46fe-cd32-3e96399e67c5: Installed remote snapshot
2024-05-07T18:35:55.759Z [WARN]  server-55663e76-e997-46fe-cd32-3e96399e67c5: skipping application of old log: index=22
2024-05-07T18:35:55.759Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: pipelining replication: peer="{Voter server-55663e76-e997-46fe-cd32-3e96399e67c5 55663e76-e997-46fe-cd32-3e96399e67c5}"
2024-05-07T18:35:55.759Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:55.759Z [INFO]  server-2efc90e0-72f9-1a01-aa60-e7520589d4b3: snapshot restore progress: id=2-10013-1715106955667 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:55.759Z [INFO]  server-2efc90e0-72f9-1a01-aa60-e7520589d4b3: removing all old logs from log store
2024-05-07T18:35:55.759Z [INFO]  server-2efc90e0-72f9-1a01-aa60-e7520589d4b3: compacting logs: from=1 to=22
2024-05-07T18:35:55.759Z [INFO]  server-2efc90e0-72f9-1a01-aa60-e7520589d4b3: Installed remote snapshot
2024-05-07T18:35:55.759Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: pipelining replication: peer="{Voter server-2efc90e0-72f9-1a01-aa60-e7520589d4b3 2efc90e0-72f9-1a01-aa60-e7520589d4b3}"
2024-05-07T18:35:56.764Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:55.759265715 +0000 UTC m=+93.360383965 (2 nodes), 1.062µs from start of poll, 1.63654893s from cluster start. Timeout at 2024-05-07 18:35:56.764306845 +0000 UTC m=+94.365425095, 1.00504113s after stability
2024-05-07T18:35:56.775Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: aborting pipeline replication: peer="{Voter server-2efc90e0-72f9-1a01-aa60-e7520589d4b3 2efc90e0-72f9-1a01-aa60-e7520589d4b3}"
2024-05-07T18:35:56.775Z [INFO]  server-75893e81-3d86-37c0-55ba-3e9c36a0908b: aborting pipeline replication: peer="{Voter server-55663e76-e997-46fe-cd32-3e96399e67c5 55663e76-e997-46fe-cd32-3e96399e67c5}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2024-05-07T18:35:56.776Z [INFO]  server-de133241-bb77-4fcc-030b-886e7a27c270: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de133241-bb77-4fcc-030b-886e7a27c270 Address:de133241-bb77-4fcc-030b-886e7a27c270} {Suffrage:Voter ID:server-73fbd892-1fb5-a33a-266f-febe665f0f61 Address:73fbd892-1fb5-a33a-266f-febe665f0f61} {Suffrage:Voter ID:server-8acbdd96-53d3-9722-ef0b-cd5c98e88d6c Address:8acbdd96-53d3-9722-ef0b-cd5c98e88d6c}]"
2024-05-07T18:35:56.776Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de133241-bb77-4fcc-030b-886e7a27c270 Address:de133241-bb77-4fcc-030b-886e7a27c270} {Suffrage:Voter ID:server-73fbd892-1fb5-a33a-266f-febe665f0f61 Address:73fbd892-1fb5-a33a-266f-febe665f0f61} {Suffrage:Voter ID:server-8acbdd96-53d3-9722-ef0b-cd5c98e88d6c Address:8acbdd96-53d3-9722-ef0b-cd5c98e88d6c}]"
2024-05-07T18:35:56.776Z [INFO]  server-8acbdd96-53d3-9722-ef0b-cd5c98e88d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de133241-bb77-4fcc-030b-886e7a27c270 Address:de133241-bb77-4fcc-030b-886e7a27c270} {Suffrage:Voter ID:server-73fbd892-1fb5-a33a-266f-febe665f0f61 Address:73fbd892-1fb5-a33a-266f-febe665f0f61} {Suffrage:Voter ID:server-8acbdd96-53d3-9722-ef0b-cd5c98e88d6c Address:8acbdd96-53d3-9722-ef0b-cd5c98e88d6c}]"
2024-05-07T18:35:56.776Z [INFO]  server-de133241-bb77-4fcc-030b-886e7a27c270: entering follower state: follower="Node at de133241-bb77-4fcc-030b-886e7a27c270 [Follower]" leader-address= leader-id=
2024-05-07T18:35:56.776Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:56.776Z [INFO]  server-8acbdd96-53d3-9722-ef0b-cd5c98e88d6c: entering follower state: follower="Node at 8acbdd96-53d3-9722-ef0b-cd5c98e88d6c [Follower]" leader-address= leader-id=
2024-05-07T18:35:56.776Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: entering follower state: follower="Node at 73fbd892-1fb5-a33a-266f-febe665f0f61 [Follower]" leader-address= leader-id=
2024-05-07T18:35:57.312Z [WARN]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:57.312Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: entering candidate state: node="Node at 73fbd892-1fb5-a33a-266f-febe665f0f61 [Candidate]" term=2
2024-05-07T18:35:57.312Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: election won: term=2 tally=2
2024-05-07T18:35:57.312Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: entering leader state: leader="Node at 73fbd892-1fb5-a33a-266f-febe665f0f61 [Leader]"
2024-05-07T18:35:57.312Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: added peer, starting replication: peer=server-de133241-bb77-4fcc-030b-886e7a27c270
2024-05-07T18:35:57.312Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: added peer, starting replication: peer=server-8acbdd96-53d3-9722-ef0b-cd5c98e88d6c
2024-05-07T18:35:57.312Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: pipelining replication: peer="{Voter server-8acbdd96-53d3-9722-ef0b-cd5c98e88d6c 8acbdd96-53d3-9722-ef0b-cd5c98e88d6c}"
2024-05-07T18:35:57.312Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: pipelining replication: peer="{Voter server-de133241-bb77-4fcc-030b-886e7a27c270 de133241-bb77-4fcc-030b-886e7a27c270}"
2024-05-07T18:35:58.317Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:57.312836018 +0000 UTC m=+94.913954279 (1 nodes), 536.138438ms from start of poll, 536.388313ms from cluster start. Timeout at 2024-05-07 18:35:58.317969451 +0000 UTC m=+95.919087701, 1.005133422s after stability
2024-05-07T18:35:58.318Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: starting snapshot up to: index=12
2024-05-07T18:35:58.318Z [INFO]  creating new snapshot: path=/tmp/raft3325895155/snapshots/2-12-1715106958318.tmp
2024-05-07T18:35:58.320Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: snapshot complete up to: index=12
2024-05-07T18:35:58.325Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4af69c-cefd-02de-22d7-ea17230dfcd7 Address:8e4af69c-cefd-02de-22d7-ea17230dfcd7} {Suffrage:Voter ID:server-31738907-21af-120e-c874-7fdb2003616d Address:31738907-21af-120e-c874-7fdb2003616d} {Suffrage:Voter ID:server-52a05b02-409e-511a-8051-3bea348e831e Address:52a05b02-409e-511a-8051-3bea348e831e}]"
2024-05-07T18:35:58.325Z [INFO]  server-31738907-21af-120e-c874-7fdb2003616d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4af69c-cefd-02de-22d7-ea17230dfcd7 Address:8e4af69c-cefd-02de-22d7-ea17230dfcd7} {Suffrage:Voter ID:server-31738907-21af-120e-c874-7fdb2003616d Address:31738907-21af-120e-c874-7fdb2003616d} {Suffrage:Voter ID:server-52a05b02-409e-511a-8051-3bea348e831e Address:52a05b02-409e-511a-8051-3bea348e831e}]"
2024-05-07T18:35:58.325Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: entering follower state: follower="Node at 8e4af69c-cefd-02de-22d7-ea17230dfcd7 [Follower]" leader-address= leader-id=
2024-05-07T18:35:58.325Z [INFO]  server-31738907-21af-120e-c874-7fdb2003616d: entering follower state: follower="Node at 31738907-21af-120e-c874-7fdb2003616d [Follower]" leader-address= leader-id=
2024-05-07T18:35:58.325Z [INFO]  server-52a05b02-409e-511a-8051-3bea348e831e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4af69c-cefd-02de-22d7-ea17230dfcd7 Address:8e4af69c-cefd-02de-22d7-ea17230dfcd7} {Suffrage:Voter ID:server-31738907-21af-120e-c874-7fdb2003616d Address:31738907-21af-120e-c874-7fdb2003616d} {Suffrage:Voter ID:server-52a05b02-409e-511a-8051-3bea348e831e Address:52a05b02-409e-511a-8051-3bea348e831e}]"
2024-05-07T18:35:58.326Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:35:58.326Z [INFO]  server-52a05b02-409e-511a-8051-3bea348e831e: entering follower state: follower="Node at 52a05b02-409e-511a-8051-3bea348e831e [Follower]" leader-address= leader-id=
2024-05-07T18:35:58.938Z [WARN]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:35:58.938Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: entering candidate state: node="Node at 8e4af69c-cefd-02de-22d7-ea17230dfcd7 [Candidate]" term=2
2024-05-07T18:35:58.938Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: election won: term=2 tally=2
2024-05-07T18:35:58.938Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: entering leader state: leader="Node at 8e4af69c-cefd-02de-22d7-ea17230dfcd7 [Leader]"
2024-05-07T18:35:58.938Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: added peer, starting replication: peer=server-31738907-21af-120e-c874-7fdb2003616d
2024-05-07T18:35:58.938Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: added peer, starting replication: peer=server-52a05b02-409e-511a-8051-3bea348e831e
2024-05-07T18:35:58.938Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: pipelining replication: peer="{Voter server-52a05b02-409e-511a-8051-3bea348e831e 52a05b02-409e-511a-8051-3bea348e831e}"
2024-05-07T18:35:58.939Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: pipelining replication: peer="{Voter server-31738907-21af-120e-c874-7fdb2003616d 31738907-21af-120e-c874-7fdb2003616d}"
2024-05-07T18:35:59.944Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:35:58.939130516 +0000 UTC m=+96.540248766 (1 nodes), 613.105583ms from start of poll, 613.43716ms from cluster start. Timeout at 2024-05-07 18:35:59.944865526 +0000 UTC m=+97.545983776, 1.00573501s after stability
2024-05-07T18:35:59.945Z [INFO]  creating new snapshot: path=/tmp/raft431805027/snapshots/2-10013-1715106959945.tmp
2024-05-07T18:35:59.945Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: copied to local snapshot: bytes=71
2024-05-07T18:35:59.945Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: snapshot restore progress: id=2-10013-1715106959945 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:59.945Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: restored user snapshot: index=10013
2024-05-07T18:35:59.945Z [ERROR] server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: failed to get log: index=3 error="log not found"
2024-05-07T18:35:59.945Z [ERROR] server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: failed to get log: index=3 error="log not found"
2024-05-07T18:35:59.945Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: aborting pipeline replication: peer="{Voter server-31738907-21af-120e-c874-7fdb2003616d 31738907-21af-120e-c874-7fdb2003616d}"
2024-05-07T18:35:59.945Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: aborting pipeline replication: peer="{Voter server-52a05b02-409e-511a-8051-3bea348e831e 52a05b02-409e-511a-8051-3bea348e831e}"
2024-05-07T18:35:59.951Z [ERROR] server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: failed to get log: index=3 error="log not found"
2024-05-07T18:35:59.952Z [INFO]  creating new snapshot: path=/tmp/raft2606233838/snapshots/2-10013-1715106959952.tmp
2024-05-07T18:35:59.955Z [ERROR] server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: failed to get log: index=3 error="log not found"
2024-05-07T18:35:59.955Z [INFO]  creating new snapshot: path=/tmp/raft4056511460/snapshots/2-10013-1715106959955.tmp
2024-05-07T18:35:59.971Z [INFO]  server-31738907-21af-120e-c874-7fdb2003616d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:59.971Z [INFO]  server-52a05b02-409e-511a-8051-3bea348e831e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:59.971Z [INFO]  server-31738907-21af-120e-c874-7fdb2003616d: copied to local snapshot: bytes=71
2024-05-07T18:35:59.971Z [INFO]  server-52a05b02-409e-511a-8051-3bea348e831e: copied to local snapshot: bytes=71
2024-05-07T18:35:59.971Z [INFO]  server-31738907-21af-120e-c874-7fdb2003616d: snapshot restore progress: id=2-10013-1715106959952 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:59.971Z [INFO]  server-31738907-21af-120e-c874-7fdb2003616d: Installed remote snapshot
2024-05-07T18:35:59.971Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: pipelining replication: peer="{Voter server-31738907-21af-120e-c874-7fdb2003616d 31738907-21af-120e-c874-7fdb2003616d}"
2024-05-07T18:35:59.971Z [INFO]  server-52a05b02-409e-511a-8051-3bea348e831e: snapshot restore progress: id=2-10013-1715106959955 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:35:59.971Z [INFO]  server-52a05b02-409e-511a-8051-3bea348e831e: Installed remote snapshot
2024-05-07T18:35:59.971Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:35:59.971Z [INFO]  server-8e4af69c-cefd-02de-22d7-ea17230dfcd7: pipelining replication: peer="{Voter server-52a05b02-409e-511a-8051-3bea348e831e 52a05b02-409e-511a-8051-3bea348e831e}"
2024-05-07T18:36:00.976Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:35:59.971581678 +0000 UTC m=+97.572699928 (2 nodes), 571ns from start of poll, 1.645888322s from cluster start. Timeout at 2024-05-07 18:36:00.976704229 +0000 UTC m=+98.577822479, 1.005122551s after stability
2024-05-07T18:36:00.989Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: aborting pipeline replication: peer="{Voter server-8acbdd96-53d3-9722-ef0b-cd5c98e88d6c 8acbdd96-53d3-9722-ef0b-cd5c98e88d6c}"
2024-05-07T18:36:00.989Z [INFO]  server-73fbd892-1fb5-a33a-266f-febe665f0f61: aborting pipeline replication: peer="{Voter server-de133241-bb77-4fcc-030b-886e7a27c270 de133241-bb77-4fcc-030b-886e7a27c270}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2024-05-07T18:36:00.989Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3de80b3-876f-887e-a45e-ba20c9e8b299 Address:b3de80b3-876f-887e-a45e-ba20c9e8b299} {Suffrage:Voter ID:server-6491bdbf-225d-ee2a-8cf9-795fd139fb2a Address:6491bdbf-225d-ee2a-8cf9-795fd139fb2a} {Suffrage:Voter ID:server-2cd60964-4024-49fa-390c-a054599a5d05 Address:2cd60964-4024-49fa-390c-a054599a5d05}]"
2024-05-07T18:36:00.990Z [INFO]  server-6491bdbf-225d-ee2a-8cf9-795fd139fb2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3de80b3-876f-887e-a45e-ba20c9e8b299 Address:b3de80b3-876f-887e-a45e-ba20c9e8b299} {Suffrage:Voter ID:server-6491bdbf-225d-ee2a-8cf9-795fd139fb2a Address:6491bdbf-225d-ee2a-8cf9-795fd139fb2a} {Suffrage:Voter ID:server-2cd60964-4024-49fa-390c-a054599a5d05 Address:2cd60964-4024-49fa-390c-a054599a5d05}]"
2024-05-07T18:36:00.990Z [INFO]  server-2cd60964-4024-49fa-390c-a054599a5d05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3de80b3-876f-887e-a45e-ba20c9e8b299 Address:b3de80b3-876f-887e-a45e-ba20c9e8b299} {Suffrage:Voter ID:server-6491bdbf-225d-ee2a-8cf9-795fd139fb2a Address:6491bdbf-225d-ee2a-8cf9-795fd139fb2a} {Suffrage:Voter ID:server-2cd60964-4024-49fa-390c-a054599a5d05 Address:2cd60964-4024-49fa-390c-a054599a5d05}]"
2024-05-07T18:36:00.990Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:00.990Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: entering follower state: follower="Node at b3de80b3-876f-887e-a45e-ba20c9e8b299 [Follower]" leader-address= leader-id=
2024-05-07T18:36:00.990Z [INFO]  server-6491bdbf-225d-ee2a-8cf9-795fd139fb2a: entering follower state: follower="Node at 6491bdbf-225d-ee2a-8cf9-795fd139fb2a [Follower]" leader-address= leader-id=
2024-05-07T18:36:00.990Z [INFO]  server-2cd60964-4024-49fa-390c-a054599a5d05: entering follower state: follower="Node at 2cd60964-4024-49fa-390c-a054599a5d05 [Follower]" leader-address= leader-id=
2024-05-07T18:36:01.822Z [WARN]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:01.822Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: entering candidate state: node="Node at b3de80b3-876f-887e-a45e-ba20c9e8b299 [Candidate]" term=2
2024-05-07T18:36:01.822Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: election won: term=2 tally=2
2024-05-07T18:36:01.822Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: entering leader state: leader="Node at b3de80b3-876f-887e-a45e-ba20c9e8b299 [Leader]"
2024-05-07T18:36:01.822Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: added peer, starting replication: peer=server-6491bdbf-225d-ee2a-8cf9-795fd139fb2a
2024-05-07T18:36:01.822Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: added peer, starting replication: peer=server-2cd60964-4024-49fa-390c-a054599a5d05
2024-05-07T18:36:01.822Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: pipelining replication: peer="{Voter server-6491bdbf-225d-ee2a-8cf9-795fd139fb2a 6491bdbf-225d-ee2a-8cf9-795fd139fb2a}"
2024-05-07T18:36:01.822Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: pipelining replication: peer="{Voter server-2cd60964-4024-49fa-390c-a054599a5d05 2cd60964-4024-49fa-390c-a054599a5d05}"
2024-05-07T18:36:02.828Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:01.822693472 +0000 UTC m=+99.423811722 (1 nodes), 832.561638ms from start of poll, 832.808507ms from cluster start. Timeout at 2024-05-07 18:36:02.828646568 +0000 UTC m=+100.429764818, 1.005953096s after stability
2024-05-07T18:36:02.831Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: starting snapshot up to: index=12
2024-05-07T18:36:02.831Z [INFO]  creating new snapshot: path=/tmp/raft3900280684/snapshots/2-12-1715106962831.tmp
2024-05-07T18:36:03.028Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: snapshot complete up to: index=12
2024-05-07T18:36:03.029Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc6bbde-0bfd-1783-35ed-391c237ed247 Address:2cc6bbde-0bfd-1783-35ed-391c237ed247} {Suffrage:Voter ID:server-23fc8c61-2801-319f-75eb-f8134a7f5d68 Address:23fc8c61-2801-319f-75eb-f8134a7f5d68} {Suffrage:Voter ID:server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e Address:c4551dd8-75e5-8d9c-1881-e6c6faeba89e}]"
2024-05-07T18:36:03.029Z [INFO]  server-23fc8c61-2801-319f-75eb-f8134a7f5d68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc6bbde-0bfd-1783-35ed-391c237ed247 Address:2cc6bbde-0bfd-1783-35ed-391c237ed247} {Suffrage:Voter ID:server-23fc8c61-2801-319f-75eb-f8134a7f5d68 Address:23fc8c61-2801-319f-75eb-f8134a7f5d68} {Suffrage:Voter ID:server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e Address:c4551dd8-75e5-8d9c-1881-e6c6faeba89e}]"
2024-05-07T18:36:03.029Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: entering follower state: follower="Node at 2cc6bbde-0bfd-1783-35ed-391c237ed247 [Follower]" leader-address= leader-id=
2024-05-07T18:36:03.029Z [INFO]  server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc6bbde-0bfd-1783-35ed-391c237ed247 Address:2cc6bbde-0bfd-1783-35ed-391c237ed247} {Suffrage:Voter ID:server-23fc8c61-2801-319f-75eb-f8134a7f5d68 Address:23fc8c61-2801-319f-75eb-f8134a7f5d68} {Suffrage:Voter ID:server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e Address:c4551dd8-75e5-8d9c-1881-e6c6faeba89e}]"
2024-05-07T18:36:03.029Z [INFO]  server-23fc8c61-2801-319f-75eb-f8134a7f5d68: entering follower state: follower="Node at 23fc8c61-2801-319f-75eb-f8134a7f5d68 [Follower]" leader-address= leader-id=
2024-05-07T18:36:03.029Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:03.029Z [INFO]  server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e: entering follower state: follower="Node at c4551dd8-75e5-8d9c-1881-e6c6faeba89e [Follower]" leader-address= leader-id=
2024-05-07T18:36:03.600Z [WARN]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:03.600Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: entering candidate state: node="Node at 2cc6bbde-0bfd-1783-35ed-391c237ed247 [Candidate]" term=2
2024-05-07T18:36:03.600Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: election won: term=2 tally=2
2024-05-07T18:36:03.600Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: entering leader state: leader="Node at 2cc6bbde-0bfd-1783-35ed-391c237ed247 [Leader]"
2024-05-07T18:36:03.600Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: added peer, starting replication: peer=server-23fc8c61-2801-319f-75eb-f8134a7f5d68
2024-05-07T18:36:03.600Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: added peer, starting replication: peer=server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e
2024-05-07T18:36:03.600Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: pipelining replication: peer="{Voter server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e c4551dd8-75e5-8d9c-1881-e6c6faeba89e}"
2024-05-07T18:36:03.600Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: pipelining replication: peer="{Voter server-23fc8c61-2801-319f-75eb-f8134a7f5d68 23fc8c61-2801-319f-75eb-f8134a7f5d68}"
2024-05-07T18:36:04.606Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:03.600853962 +0000 UTC m=+101.201972212 (1 nodes), 571.123188ms from start of poll, 571.431922ms from cluster start. Timeout at 2024-05-07 18:36:04.605995498 +0000 UTC m=+102.207113758, 1.005141546s after stability
2024-05-07T18:36:04.689Z [INFO]  creating new snapshot: path=/tmp/raft3866590934/snapshots/2-10013-1715106964689.tmp
2024-05-07T18:36:04.706Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: copied to local snapshot: bytes=71
2024-05-07T18:36:04.706Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: snapshot restore progress: id=2-10013-1715106964689 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:36:04.706Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: removing all old logs from log store
2024-05-07T18:36:04.706Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: compacting logs: from=1 to=2
2024-05-07T18:36:04.706Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: restored user snapshot: index=10013
2024-05-07T18:36:04.707Z [ERROR] server-2cc6bbde-0bfd-1783-35ed-391c237ed247: failed to get log: index=2 error="log not found"
2024-05-07T18:36:04.707Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: aborting pipeline replication: peer="{Voter server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e c4551dd8-75e5-8d9c-1881-e6c6faeba89e}"
2024-05-07T18:36:04.707Z [ERROR] server-2cc6bbde-0bfd-1783-35ed-391c237ed247: failed to get log: index=2 error="log not found"
2024-05-07T18:36:04.707Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: aborting pipeline replication: peer="{Voter server-23fc8c61-2801-319f-75eb-f8134a7f5d68 23fc8c61-2801-319f-75eb-f8134a7f5d68}"
2024-05-07T18:36:04.715Z [ERROR] server-2cc6bbde-0bfd-1783-35ed-391c237ed247: failed to get log: index=2 error="log not found"
2024-05-07T18:36:04.715Z [INFO]  creating new snapshot: path=/tmp/raft19949600/snapshots/2-10013-1715106964715.tmp
2024-05-07T18:36:04.715Z [INFO]  server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:36:04.717Z [ERROR] server-2cc6bbde-0bfd-1783-35ed-391c237ed247: failed to get log: index=2 error="log not found"
2024-05-07T18:36:04.717Z [INFO]  creating new snapshot: path=/tmp/raft3443067457/snapshots/2-10013-1715106964717.tmp
2024-05-07T18:36:04.717Z [INFO]  server-23fc8c61-2801-319f-75eb-f8134a7f5d68: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-05-07T18:36:04.941Z [INFO]  server-23fc8c61-2801-319f-75eb-f8134a7f5d68: copied to local snapshot: bytes=71
2024-05-07T18:36:04.941Z [INFO]  server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e: copied to local snapshot: bytes=71
2024-05-07T18:36:04.941Z [INFO]  server-23fc8c61-2801-319f-75eb-f8134a7f5d68: snapshot restore progress: id=2-10013-1715106964717 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:36:04.941Z [INFO]  server-23fc8c61-2801-319f-75eb-f8134a7f5d68: removing all old logs from log store
2024-05-07T18:36:04.941Z [INFO]  server-23fc8c61-2801-319f-75eb-f8134a7f5d68: compacting logs: from=1 to=2
2024-05-07T18:36:04.941Z [INFO]  server-23fc8c61-2801-319f-75eb-f8134a7f5d68: Installed remote snapshot
2024-05-07T18:36:04.941Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: pipelining replication: peer="{Voter server-23fc8c61-2801-319f-75eb-f8134a7f5d68 23fc8c61-2801-319f-75eb-f8134a7f5d68}"
2024-05-07T18:36:04.941Z [INFO]  server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e: snapshot restore progress: id=2-10013-1715106964715 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-05-07T18:36:04.941Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:04.941Z [INFO]  server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e: removing all old logs from log store
2024-05-07T18:36:04.941Z [INFO]  server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e: compacting logs: from=1 to=2
2024-05-07T18:36:04.941Z [INFO]  server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e: Installed remote snapshot
2024-05-07T18:36:04.942Z [INFO]  server-2cc6bbde-0bfd-1783-35ed-391c237ed247: pipelining replication: peer="{Voter server-c4551dd8-75e5-8d9c-1881-e6c6faeba89e c4551dd8-75e5-8d9c-1881-e6c6faeba89e}"
2024-05-07T18:36:05.947Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:04.941963661 +0000 UTC m=+102.543081911 (2 nodes), 391ns from start of poll, 1.912541621s from cluster start. Timeout at 2024-05-07 18:36:05.947101631 +0000 UTC m=+103.548219881, 1.00513797s after stability
2024-05-07T18:36:05.957Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: aborting pipeline replication: peer="{Voter server-2cd60964-4024-49fa-390c-a054599a5d05 2cd60964-4024-49fa-390c-a054599a5d05}"
2024-05-07T18:36:05.958Z [INFO]  server-b3de80b3-876f-887e-a45e-ba20c9e8b299: aborting pipeline replication: peer="{Voter server-6491bdbf-225d-ee2a-8cf9-795fd139fb2a 6491bdbf-225d-ee2a-8cf9-795fd139fb2a}"
--- PASS: TestRaft_UserRestore (88.17s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.62s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.71s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.34s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.48s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (3.17s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.80s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.35s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.66s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.93s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.37s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.32s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.89s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.76s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (5.01s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.34s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.87s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.89s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.74s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.47s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (3.18s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.65s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.21s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.97s)
=== RUN   TestRaft_SendSnapshotFollower
2024-05-07T18:36:05.959Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2555e1b4-cc14-a16c-7113-41b4722c93ae Address:2555e1b4-cc14-a16c-7113-41b4722c93ae} {Suffrage:Voter ID:server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d Address:7199a408-ef4e-9fe7-eaf3-09548aa3b04d} {Suffrage:Voter ID:server-972b854a-9b42-f109-202f-aeb77160442b Address:972b854a-9b42-f109-202f-aeb77160442b}]"
2024-05-07T18:36:05.959Z [INFO]  server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2555e1b4-cc14-a16c-7113-41b4722c93ae Address:2555e1b4-cc14-a16c-7113-41b4722c93ae} {Suffrage:Voter ID:server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d Address:7199a408-ef4e-9fe7-eaf3-09548aa3b04d} {Suffrage:Voter ID:server-972b854a-9b42-f109-202f-aeb77160442b Address:972b854a-9b42-f109-202f-aeb77160442b}]"
2024-05-07T18:36:05.959Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2555e1b4-cc14-a16c-7113-41b4722c93ae Address:2555e1b4-cc14-a16c-7113-41b4722c93ae} {Suffrage:Voter ID:server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d Address:7199a408-ef4e-9fe7-eaf3-09548aa3b04d} {Suffrage:Voter ID:server-972b854a-9b42-f109-202f-aeb77160442b Address:972b854a-9b42-f109-202f-aeb77160442b}]"
2024-05-07T18:36:05.959Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:05.959Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: entering follower state: follower="Node at 2555e1b4-cc14-a16c-7113-41b4722c93ae [Follower]" leader-address= leader-id=
2024-05-07T18:36:05.959Z [INFO]  server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d: entering follower state: follower="Node at 7199a408-ef4e-9fe7-eaf3-09548aa3b04d [Follower]" leader-address= leader-id=
2024-05-07T18:36:05.959Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: entering follower state: follower="Node at 972b854a-9b42-f109-202f-aeb77160442b [Follower]" leader-address= leader-id=
2024-05-07T18:36:06.014Z [WARN]  server-972b854a-9b42-f109-202f-aeb77160442b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:06.014Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: entering candidate state: node="Node at 972b854a-9b42-f109-202f-aeb77160442b [Candidate]" term=2
2024-05-07T18:36:06.014Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: election won: term=2 tally=2
2024-05-07T18:36:06.014Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: entering leader state: leader="Node at 972b854a-9b42-f109-202f-aeb77160442b [Leader]"
2024-05-07T18:36:06.014Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: added peer, starting replication: peer=server-2555e1b4-cc14-a16c-7113-41b4722c93ae
2024-05-07T18:36:06.014Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: added peer, starting replication: peer=server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d
2024-05-07T18:36:06.014Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: pipelining replication: peer="{Voter server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d 7199a408-ef4e-9fe7-eaf3-09548aa3b04d}"
2024-05-07T18:36:06.014Z [WARN]  server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d: rejecting vote request since our last term is greater: candidate=972b854a-9b42-f109-202f-aeb77160442b last-term=2 last-candidate-term=1
2024-05-07T18:36:06.014Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: pipelining replication: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}"
2024-05-07T18:36:06.119Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:06.014554139 +0000 UTC m=+103.615672389 (2 nodes), 55.104752ms from start of poll, 55.396114ms from cluster start. Timeout at 2024-05-07 18:36:06.119660862 +0000 UTC m=+103.720779112, 105.106723ms after stability
2024-05-07T18:36:06.119Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:06.225Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:06.119723508 +0000 UTC m=+103.720841758 (1 nodes), 1.072µs from start of poll, 160.565483ms from cluster start. Timeout at 2024-05-07 18:36:06.225294404 +0000 UTC m=+103.826412654, 105.570896ms after stability
2024-05-07T18:36:06.225Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to pipeline appendEntries: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}" error="append pipeline closed"
2024-05-07T18:36:06.225Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: aborting pipeline replication: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}"
2024-05-07T18:36:06.225Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to appendEntries to: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}" error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2024-05-07T18:36:06.227Z [ERROR] server-2555e1b4-cc14-a16c-7113-41b4722c93ae: failed to take snapshot: error="nothing new to snapshot"
2024-05-07T18:36:06.227Z [INFO]  server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d: starting snapshot up to: index=97
2024-05-07T18:36:06.227Z [INFO]  creating new snapshot: path=/tmp/raft3896969916/snapshots/2-97-1715106966227.tmp
2024-05-07T18:36:06.231Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to heartbeat to: peer=2555e1b4-cc14-a16c-7113-41b4722c93ae backoff time=10ms error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
2024-05-07T18:36:06.236Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to appendEntries to: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}" error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
2024-05-07T18:36:06.246Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to appendEntries to: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}" error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
2024-05-07T18:36:06.254Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to heartbeat to: peer=2555e1b4-cc14-a16c-7113-41b4722c93ae backoff time=10ms error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
2024-05-07T18:36:06.273Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to heartbeat to: peer=2555e1b4-cc14-a16c-7113-41b4722c93ae backoff time=10ms error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
2024-05-07T18:36:06.273Z [WARN]  server-972b854a-9b42-f109-202f-aeb77160442b: failed to contact: server-id=server-2555e1b4-cc14-a16c-7113-41b4722c93ae time=50.789085ms
2024-05-07T18:36:06.276Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to appendEntries to: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}" error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
2024-05-07T18:36:06.284Z [WARN]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: heartbeat timeout reached, starting election: last-leader-addr=972b854a-9b42-f109-202f-aeb77160442b last-leader-id=server-972b854a-9b42-f109-202f-aeb77160442b
2024-05-07T18:36:06.284Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: entering candidate state: node="Node at 2555e1b4-cc14-a16c-7113-41b4722c93ae [Candidate]" term=3
2024-05-07T18:36:06.284Z [ERROR] server-2555e1b4-cc14-a16c-7113-41b4722c93ae: failed to make requestVote RPC: target="{Voter server-972b854a-9b42-f109-202f-aeb77160442b 972b854a-9b42-f109-202f-aeb77160442b}" error="failed to connect to peer: 972b854a-9b42-f109-202f-aeb77160442b" term=3
2024-05-07T18:36:06.284Z [ERROR] server-2555e1b4-cc14-a16c-7113-41b4722c93ae: failed to make requestVote RPC: target="{Voter server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d 7199a408-ef4e-9fe7-eaf3-09548aa3b04d}" error="failed to connect to peer: 7199a408-ef4e-9fe7-eaf3-09548aa3b04d" term=3
2024-05-07T18:36:06.291Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to heartbeat to: peer=2555e1b4-cc14-a16c-7113-41b4722c93ae backoff time=20ms error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
2024-05-07T18:36:06.321Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to heartbeat to: peer=2555e1b4-cc14-a16c-7113-41b4722c93ae backoff time=25ms error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
2024-05-07T18:36:06.323Z [WARN]  server-972b854a-9b42-f109-202f-aeb77160442b: failed to contact: server-id=server-2555e1b4-cc14-a16c-7113-41b4722c93ae time=100.837743ms
2024-05-07T18:36:06.327Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to appendEntries to: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}" error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
2024-05-07T18:36:06.356Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to heartbeat to: peer=2555e1b4-cc14-a16c-7113-41b4722c93ae backoff time=25ms error="failed to connect to peer: 2555e1b4-cc14-a16c-7113-41b4722c93ae"
2024-05-07T18:36:06.371Z [WARN]  server-972b854a-9b42-f109-202f-aeb77160442b: failed to contact: server-id=server-2555e1b4-cc14-a16c-7113-41b4722c93ae time=148.437897ms
2024-05-07T18:36:06.371Z [WARN]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: Election timeout reached, restarting election
2024-05-07T18:36:06.371Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: entering candidate state: node="Node at 2555e1b4-cc14-a16c-7113-41b4722c93ae [Candidate]" term=4
2024-05-07T18:36:06.371Z [ERROR] server-2555e1b4-cc14-a16c-7113-41b4722c93ae: failed to make requestVote RPC: target="{Voter server-972b854a-9b42-f109-202f-aeb77160442b 972b854a-9b42-f109-202f-aeb77160442b}" error="failed to connect to peer: 972b854a-9b42-f109-202f-aeb77160442b" term=4
2024-05-07T18:36:06.371Z [ERROR] server-2555e1b4-cc14-a16c-7113-41b4722c93ae: failed to make requestVote RPC: target="{Voter server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d 7199a408-ef4e-9fe7-eaf3-09548aa3b04d}" error="failed to connect to peer: 7199a408-ef4e-9fe7-eaf3-09548aa3b04d" term=4
2024-05-07T18:36:06.383Z [INFO]  server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d: compacting logs: from=1 to=92
2024-05-07T18:36:06.383Z [INFO]  server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d: snapshot complete up to: index=97
2024-05-07T18:36:06.383Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: starting snapshot up to: index=102
2024-05-07T18:36:06.383Z [INFO]  creating new snapshot: path=/tmp/raft2153083539/snapshots/2-102-1715106966383.tmp
2024-05-07T18:36:06.384Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: compacting logs: from=1 to=92
2024-05-07T18:36:06.384Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: snapshot complete up to: index=102
2024-05-07T18:36:06.413Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to get log: index=2 error="log not found"
2024-05-07T18:36:06.413Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: ignoring installSnapshot request with older term than current term: request-term=2 current-term=4
2024-05-07T18:36:06.413Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: peer has newer term, stopping replication: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}"
2024-05-07T18:36:06.413Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: entering follower state: follower="Node at 972b854a-9b42-f109-202f-aeb77160442b [Follower]" leader-address= leader-id=
2024-05-07T18:36:06.413Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: aborting pipeline replication: peer="{Voter server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d 7199a408-ef4e-9fe7-eaf3-09548aa3b04d}"
2024-05-07T18:36:06.470Z [WARN]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: Election timeout reached, restarting election
2024-05-07T18:36:06.470Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: entering candidate state: node="Node at 2555e1b4-cc14-a16c-7113-41b4722c93ae [Candidate]" term=5
2024-05-07T18:36:06.470Z [WARN]  server-972b854a-9b42-f109-202f-aeb77160442b: rejecting vote request since our last index is greater: candidate=2555e1b4-cc14-a16c-7113-41b4722c93ae last-index=102 last-candidate-index=2
2024-05-07T18:36:06.470Z [WARN]  server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d: rejecting vote request since we have a leader: from=2555e1b4-cc14-a16c-7113-41b4722c93ae leader=972b854a-9b42-f109-202f-aeb77160442b leader-id=server-972b854a-9b42-f109-202f-aeb77160442b
2024-05-07T18:36:06.484Z [WARN]  server-972b854a-9b42-f109-202f-aeb77160442b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:06.484Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: entering candidate state: node="Node at 972b854a-9b42-f109-202f-aeb77160442b [Candidate]" term=6
2024-05-07T18:36:06.484Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: election won: term=6 tally=2
2024-05-07T18:36:06.484Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: entering leader state: leader="Node at 972b854a-9b42-f109-202f-aeb77160442b [Leader]"
2024-05-07T18:36:06.484Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: added peer, starting replication: peer=server-2555e1b4-cc14-a16c-7113-41b4722c93ae
2024-05-07T18:36:06.484Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: added peer, starting replication: peer=server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d
2024-05-07T18:36:06.484Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: pipelining replication: peer="{Voter server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d 7199a408-ef4e-9fe7-eaf3-09548aa3b04d}"
2024-05-07T18:36:06.484Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: entering follower state: follower="Node at 2555e1b4-cc14-a16c-7113-41b4722c93ae [Follower]" leader-address= leader-id=
2024-05-07T18:36:06.484Z [WARN]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: failed to get previous log: previous-index=102 last-index=2 error="log not found"
2024-05-07T18:36:06.484Z [WARN]  server-972b854a-9b42-f109-202f-aeb77160442b: appendEntries rejected, sending older logs: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}" next=3
2024-05-07T18:36:06.484Z [ERROR] server-972b854a-9b42-f109-202f-aeb77160442b: failed to get log: index=2 error="log not found"
2024-05-07T18:36:06.484Z [INFO]  creating new snapshot: path=/tmp/raft598515935/snapshots/2-102-1715106966484.tmp
2024-05-07T18:36:06.534Z [WARN]  server-972b854a-9b42-f109-202f-aeb77160442b: failed to contact: server-id=server-2555e1b4-cc14-a16c-7113-41b4722c93ae time=50.137934ms
2024-05-07T18:36:06.584Z [WARN]  server-972b854a-9b42-f109-202f-aeb77160442b: failed to contact: server-id=server-2555e1b4-cc14-a16c-7113-41b4722c93ae time=99.415707ms
2024-05-07T18:36:06.632Z [WARN]  server-972b854a-9b42-f109-202f-aeb77160442b: failed to contact: server-id=server-2555e1b4-cc14-a16c-7113-41b4722c93ae time=147.708028ms
2024-05-07T18:36:06.642Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2024-05-07T18:36:06.642Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: copied to local snapshot: bytes=693
2024-05-07T18:36:06.642Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: snapshot restore progress: id=2-102-1715106966484 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2024-05-07T18:36:06.642Z [INFO]  server-2555e1b4-cc14-a16c-7113-41b4722c93ae: Installed remote snapshot
2024-05-07T18:36:06.642Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: pipelining replication: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}"
2024-05-07T18:36:06.647Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: aborting pipeline replication: peer="{Voter server-7199a408-ef4e-9fe7-eaf3-09548aa3b04d 7199a408-ef4e-9fe7-eaf3-09548aa3b04d}"
2024-05-07T18:36:06.647Z [INFO]  server-972b854a-9b42-f109-202f-aeb77160442b: aborting pipeline replication: peer="{Voter server-2555e1b4-cc14-a16c-7113-41b4722c93ae 2555e1b4-cc14-a16c-7113-41b4722c93ae}"
--- PASS: TestRaft_SendSnapshotFollower (0.69s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2024-05-07T18:36:06.648Z [INFO]  server-e56d085c-d4cd-5330-b160-c162b65c6441: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56d085c-d4cd-5330-b160-c162b65c6441 Address:e56d085c-d4cd-5330-b160-c162b65c6441} {Suffrage:Voter ID:server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39 Address:f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39} {Suffrage:Voter ID:server-33ddbdfa-2eed-a432-5eb4-55760c278021 Address:33ddbdfa-2eed-a432-5eb4-55760c278021}]"
2024-05-07T18:36:06.649Z [INFO]  server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56d085c-d4cd-5330-b160-c162b65c6441 Address:e56d085c-d4cd-5330-b160-c162b65c6441} {Suffrage:Voter ID:server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39 Address:f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39} {Suffrage:Voter ID:server-33ddbdfa-2eed-a432-5eb4-55760c278021 Address:33ddbdfa-2eed-a432-5eb4-55760c278021}]"
2024-05-07T18:36:06.649Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56d085c-d4cd-5330-b160-c162b65c6441 Address:e56d085c-d4cd-5330-b160-c162b65c6441} {Suffrage:Voter ID:server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39 Address:f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39} {Suffrage:Voter ID:server-33ddbdfa-2eed-a432-5eb4-55760c278021 Address:33ddbdfa-2eed-a432-5eb4-55760c278021}]"
2024-05-07T18:36:06.649Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:06.649Z [INFO]  server-e56d085c-d4cd-5330-b160-c162b65c6441: entering follower state: follower="Node at e56d085c-d4cd-5330-b160-c162b65c6441 [Follower]" leader-address= leader-id=
2024-05-07T18:36:06.649Z [INFO]  server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39: entering follower state: follower="Node at f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39 [Follower]" leader-address= leader-id=
2024-05-07T18:36:06.649Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: entering follower state: follower="Node at 33ddbdfa-2eed-a432-5eb4-55760c278021 [Follower]" leader-address= leader-id=
2024-05-07T18:36:06.701Z [WARN]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:06.701Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: entering candidate state: node="Node at 33ddbdfa-2eed-a432-5eb4-55760c278021 [Candidate]" term=2
2024-05-07T18:36:06.701Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: election won: term=2 tally=2
2024-05-07T18:36:06.701Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: entering leader state: leader="Node at 33ddbdfa-2eed-a432-5eb4-55760c278021 [Leader]"
2024-05-07T18:36:06.701Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: added peer, starting replication: peer=server-e56d085c-d4cd-5330-b160-c162b65c6441
2024-05-07T18:36:06.701Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: added peer, starting replication: peer=server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39
2024-05-07T18:36:06.701Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: pipelining replication: peer="{Voter server-e56d085c-d4cd-5330-b160-c162b65c6441 e56d085c-d4cd-5330-b160-c162b65c6441}"
2024-05-07T18:36:06.701Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: pipelining replication: peer="{Voter server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39 f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39}"
2024-05-07T18:36:06.806Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:06.701192238 +0000 UTC m=+104.302310488 (2 nodes), 52.043218ms from start of poll, 52.305856ms from cluster start. Timeout at 2024-05-07 18:36:06.806381103 +0000 UTC m=+104.407499353, 105.188865ms after stability
2024-05-07T18:36:06.806Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:06.912Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:06.80643871 +0000 UTC m=+104.407556960 (1 nodes), 1.443µs from start of poll, 157.552328ms from cluster start. Timeout at 2024-05-07 18:36:06.91203871 +0000 UTC m=+104.513156960, 105.6ms after stability
2024-05-07T18:36:06.912Z [ERROR] server-33ddbdfa-2eed-a432-5eb4-55760c278021: failed to pipeline appendEntries: peer="{Voter server-e56d085c-d4cd-5330-b160-c162b65c6441 e56d085c-d4cd-5330-b160-c162b65c6441}" error="append pipeline closed"
2024-05-07T18:36:06.912Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: aborting pipeline replication: peer="{Voter server-e56d085c-d4cd-5330-b160-c162b65c6441 e56d085c-d4cd-5330-b160-c162b65c6441}"
2024-05-07T18:36:06.912Z [ERROR] server-33ddbdfa-2eed-a432-5eb4-55760c278021: failed to appendEntries to: peer="{Voter server-e56d085c-d4cd-5330-b160-c162b65c6441 e56d085c-d4cd-5330-b160-c162b65c6441}" error="failed to connect to peer: e56d085c-d4cd-5330-b160-c162b65c6441"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2024-05-07T18:36:06.915Z [ERROR] server-e56d085c-d4cd-5330-b160-c162b65c6441: failed to take snapshot: error="nothing new to snapshot"
2024-05-07T18:36:06.915Z [INFO]  server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39: starting snapshot up to: index=100
2024-05-07T18:36:06.915Z [INFO]  creating new snapshot: path=/tmp/raft660992777/snapshots/2-100-1715106966915.tmp
2024-05-07T18:36:06.918Z [ERROR] server-33ddbdfa-2eed-a432-5eb4-55760c278021: failed to heartbeat to: peer=e56d085c-d4cd-5330-b160-c162b65c6441 backoff time=10ms error="failed to connect to peer: e56d085c-d4cd-5330-b160-c162b65c6441"
2024-05-07T18:36:06.922Z [ERROR] server-33ddbdfa-2eed-a432-5eb4-55760c278021: failed to appendEntries to: peer="{Voter server-e56d085c-d4cd-5330-b160-c162b65c6441 e56d085c-d4cd-5330-b160-c162b65c6441}" error="failed to connect to peer: e56d085c-d4cd-5330-b160-c162b65c6441"
2024-05-07T18:36:06.923Z [INFO]  server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39: compacting logs: from=1 to=92
2024-05-07T18:36:06.923Z [INFO]  server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39: snapshot complete up to: index=100
2024-05-07T18:36:06.923Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: starting snapshot up to: index=102
2024-05-07T18:36:06.923Z [INFO]  creating new snapshot: path=/tmp/raft3585126083/snapshots/2-102-1715106966923.tmp
2024-05-07T18:36:06.924Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: compacting logs: from=1 to=92
2024-05-07T18:36:06.924Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: snapshot complete up to: index=102
    raft_test.go:1690: [INFO] Finished apply without behind follower
2024-05-07T18:36:06.933Z [ERROR] server-33ddbdfa-2eed-a432-5eb4-55760c278021: failed to get log: index=2 error="log not found"
2024-05-07T18:36:07.171Z [INFO]  creating new snapshot: path=/tmp/raft1077995944/snapshots/2-102-1715106967171.tmp
2024-05-07T18:36:07.171Z [INFO]  server-e56d085c-d4cd-5330-b160-c162b65c6441: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2024-05-07T18:36:07.172Z [INFO]  server-e56d085c-d4cd-5330-b160-c162b65c6441: copied to local snapshot: bytes=693
2024-05-07T18:36:07.172Z [INFO]  server-e56d085c-d4cd-5330-b160-c162b65c6441: snapshot restore progress: id=2-102-1715106967171 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2024-05-07T18:36:07.172Z [INFO]  server-e56d085c-d4cd-5330-b160-c162b65c6441: Installed remote snapshot
2024-05-07T18:36:07.172Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: pipelining replication: peer="{Voter server-e56d085c-d4cd-5330-b160-c162b65c6441 e56d085c-d4cd-5330-b160-c162b65c6441}"
2024-05-07T18:36:07.176Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: aborting pipeline replication: peer="{Voter server-f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39 f6037f6d-8134-ebd9-cfcf-9acd4e1c0f39}"
2024-05-07T18:36:07.176Z [INFO]  server-33ddbdfa-2eed-a432-5eb4-55760c278021: aborting pipeline replication: peer="{Voter server-e56d085c-d4cd-5330-b160-c162b65c6441 e56d085c-d4cd-5330-b160-c162b65c6441}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.54s)
=== RUN   TestRaft_ReJoinFollower
2024-05-07T18:36:07.192Z [INFO]  server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 Address:63bbe9d7-4863-1e66-cff2-4312d5ea0dd4} {Suffrage:Voter ID:server-76ddfc92-a982-7f35-3186-1cf510f9fb25 Address:76ddfc92-a982-7f35-3186-1cf510f9fb25} {Suffrage:Voter ID:server-c62b02e2-818a-4141-454d-4564541924ae Address:c62b02e2-818a-4141-454d-4564541924ae}]"
2024-05-07T18:36:07.192Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 Address:63bbe9d7-4863-1e66-cff2-4312d5ea0dd4} {Suffrage:Voter ID:server-76ddfc92-a982-7f35-3186-1cf510f9fb25 Address:76ddfc92-a982-7f35-3186-1cf510f9fb25} {Suffrage:Voter ID:server-c62b02e2-818a-4141-454d-4564541924ae Address:c62b02e2-818a-4141-454d-4564541924ae}]"
2024-05-07T18:36:07.192Z [INFO]  server-c62b02e2-818a-4141-454d-4564541924ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 Address:63bbe9d7-4863-1e66-cff2-4312d5ea0dd4} {Suffrage:Voter ID:server-76ddfc92-a982-7f35-3186-1cf510f9fb25 Address:76ddfc92-a982-7f35-3186-1cf510f9fb25} {Suffrage:Voter ID:server-c62b02e2-818a-4141-454d-4564541924ae Address:c62b02e2-818a-4141-454d-4564541924ae}]"
2024-05-07T18:36:07.192Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:07.192Z [INFO]  server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4: entering follower state: follower="Node at 63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 [Follower]" leader-address= leader-id=
2024-05-07T18:36:07.192Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: entering follower state: follower="Node at 76ddfc92-a982-7f35-3186-1cf510f9fb25 [Follower]" leader-address= leader-id=
2024-05-07T18:36:07.192Z [INFO]  server-c62b02e2-818a-4141-454d-4564541924ae: entering follower state: follower="Node at c62b02e2-818a-4141-454d-4564541924ae [Follower]" leader-address= leader-id=
2024-05-07T18:36:07.249Z [WARN]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:07.249Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: entering candidate state: node="Node at 76ddfc92-a982-7f35-3186-1cf510f9fb25 [Candidate]" term=2
2024-05-07T18:36:07.249Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: election won: term=2 tally=2
2024-05-07T18:36:07.249Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: entering leader state: leader="Node at 76ddfc92-a982-7f35-3186-1cf510f9fb25 [Leader]"
2024-05-07T18:36:07.249Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: added peer, starting replication: peer=server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4
2024-05-07T18:36:07.249Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: added peer, starting replication: peer=server-c62b02e2-818a-4141-454d-4564541924ae
2024-05-07T18:36:07.249Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: pipelining replication: peer="{Voter server-c62b02e2-818a-4141-454d-4564541924ae c62b02e2-818a-4141-454d-4564541924ae}"
2024-05-07T18:36:07.249Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: pipelining replication: peer="{Voter server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 63bbe9d7-4863-1e66-cff2-4312d5ea0dd4}"
2024-05-07T18:36:07.354Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:07.249834331 +0000 UTC m=+104.850952581 (1 nodes), 57.506318ms from start of poll, 57.735274ms from cluster start. Timeout at 2024-05-07 18:36:07.354841067 +0000 UTC m=+104.955959317, 105.006736ms after stability
2024-05-07T18:36:07.360Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:07.466Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:07.360862026 +0000 UTC m=+104.961980276 (2 nodes), 1.533µs from start of poll, 168.762969ms from cluster start. Timeout at 2024-05-07 18:36:07.466043268 +0000 UTC m=+105.067161518, 105.181242ms after stability
2024-05-07T18:36:07.471Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: updating configuration: command=RemoveServer server-id=server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 server-addr= servers="[{Suffrage:Voter ID:server-76ddfc92-a982-7f35-3186-1cf510f9fb25 Address:76ddfc92-a982-7f35-3186-1cf510f9fb25} {Suffrage:Voter ID:server-c62b02e2-818a-4141-454d-4564541924ae Address:c62b02e2-818a-4141-454d-4564541924ae}]"
2024-05-07T18:36:07.471Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: removed peer, stopping replication: peer=server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 last-index=3
2024-05-07T18:36:07.471Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: aborting pipeline replication: peer="{Voter server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 63bbe9d7-4863-1e66-cff2-4312d5ea0dd4}"
2024-05-07T18:36:07.543Z [WARN]  server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2024-05-07T18:36:07.577Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: updating configuration: command=AddVoter server-id=server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 server-addr=63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 servers="[{Suffrage:Voter ID:server-76ddfc92-a982-7f35-3186-1cf510f9fb25 Address:76ddfc92-a982-7f35-3186-1cf510f9fb25} {Suffrage:Voter ID:server-c62b02e2-818a-4141-454d-4564541924ae Address:c62b02e2-818a-4141-454d-4564541924ae} {Suffrage:Voter ID:server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 Address:63bbe9d7-4863-1e66-cff2-4312d5ea0dd4}]"
2024-05-07T18:36:07.577Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: added peer, starting replication: peer=server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4
2024-05-07T18:36:07.577Z [WARN]  server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2024-05-07T18:36:07.577Z [WARN]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: appendEntries rejected, sending older logs: peer="{Voter server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 63bbe9d7-4863-1e66-cff2-4312d5ea0dd4}" next=4
2024-05-07T18:36:07.577Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: pipelining replication: peer="{Voter server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 63bbe9d7-4863-1e66-cff2-4312d5ea0dd4}"
2024-05-07T18:36:07.577Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:07.683Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:07.577607841 +0000 UTC m=+105.178726081 (1 nodes), 300ns from start of poll, 385.508774ms from cluster start. Timeout at 2024-05-07 18:36:07.683328015 +0000 UTC m=+105.284446275, 105.720194ms after stability
2024-05-07T18:36:07.683Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: aborting pipeline replication: peer="{Voter server-c62b02e2-818a-4141-454d-4564541924ae c62b02e2-818a-4141-454d-4564541924ae}"
2024-05-07T18:36:07.683Z [INFO]  server-76ddfc92-a982-7f35-3186-1cf510f9fb25: aborting pipeline replication: peer="{Voter server-63bbe9d7-4863-1e66-cff2-4312d5ea0dd4 63bbe9d7-4863-1e66-cff2-4312d5ea0dd4}"
--- PASS: TestRaft_ReJoinFollower (0.58s)
=== RUN   TestRaft_LeaderLeaseExpire
2024-05-07T18:36:07.775Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a15824cc-2f30-201a-6047-b1755b6ab586 Address:a15824cc-2f30-201a-6047-b1755b6ab586} {Suffrage:Voter ID:server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc Address:344ed926-d5b6-7c2d-14e5-1060ccf51fdc}]"
2024-05-07T18:36:07.776Z [INFO]  server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a15824cc-2f30-201a-6047-b1755b6ab586 Address:a15824cc-2f30-201a-6047-b1755b6ab586} {Suffrage:Voter ID:server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc Address:344ed926-d5b6-7c2d-14e5-1060ccf51fdc}]"
2024-05-07T18:36:07.775Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: entering follower state: follower="Node at a15824cc-2f30-201a-6047-b1755b6ab586 [Follower]" leader-address= leader-id=
2024-05-07T18:36:07.776Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:07.776Z [INFO]  server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: entering follower state: follower="Node at 344ed926-d5b6-7c2d-14e5-1060ccf51fdc [Follower]" leader-address= leader-id=
2024-05-07T18:36:07.837Z [WARN]  server-a15824cc-2f30-201a-6047-b1755b6ab586: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:07.837Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: entering candidate state: node="Node at a15824cc-2f30-201a-6047-b1755b6ab586 [Candidate]" term=2
2024-05-07T18:36:07.837Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: election won: term=2 tally=2
2024-05-07T18:36:07.837Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: entering leader state: leader="Node at a15824cc-2f30-201a-6047-b1755b6ab586 [Leader]"
2024-05-07T18:36:07.837Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: added peer, starting replication: peer=server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc
2024-05-07T18:36:07.837Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: pipelining replication: peer="{Voter server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc 344ed926-d5b6-7c2d-14e5-1060ccf51fdc}"
2024-05-07T18:36:07.942Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:07.837501229 +0000 UTC m=+105.438619479 (1 nodes), 61.441147ms from start of poll, 61.694158ms from cluster start. Timeout at 2024-05-07 18:36:07.942715301 +0000 UTC m=+105.543833551, 105.214072ms after stability
2024-05-07T18:36:07.947Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:08.053Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:07.947828812 +0000 UTC m=+105.548947062 (1 nodes), 1.683µs from start of poll, 172.021741ms from cluster start. Timeout at 2024-05-07 18:36:08.053619757 +0000 UTC m=+105.654738017, 105.790955ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 344ed926-d5b6-7c2d-14e5-1060ccf51fdc [Follower]
2024-05-07T18:36:08.054Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to heartbeat to: peer=344ed926-d5b6-7c2d-14e5-1060ccf51fdc backoff time=10ms error="failed to connect to peer: 344ed926-d5b6-7c2d-14e5-1060ccf51fdc"
2024-05-07T18:36:08.061Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to pipeline appendEntries: peer="{Voter server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc 344ed926-d5b6-7c2d-14e5-1060ccf51fdc}" error="append pipeline closed"
2024-05-07T18:36:08.061Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: aborting pipeline replication: peer="{Voter server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc 344ed926-d5b6-7c2d-14e5-1060ccf51fdc}"
2024-05-07T18:36:08.070Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to appendEntries to: peer="{Voter server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc 344ed926-d5b6-7c2d-14e5-1060ccf51fdc}" error="failed to connect to peer: 344ed926-d5b6-7c2d-14e5-1060ccf51fdc"
2024-05-07T18:36:08.071Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to heartbeat to: peer=344ed926-d5b6-7c2d-14e5-1060ccf51fdc backoff time=10ms error="failed to connect to peer: 344ed926-d5b6-7c2d-14e5-1060ccf51fdc"
2024-05-07T18:36:08.088Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to heartbeat to: peer=344ed926-d5b6-7c2d-14e5-1060ccf51fdc backoff time=10ms error="failed to connect to peer: 344ed926-d5b6-7c2d-14e5-1060ccf51fdc"
2024-05-07T18:36:08.088Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to appendEntries to: peer="{Voter server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc 344ed926-d5b6-7c2d-14e5-1060ccf51fdc}" error="failed to connect to peer: 344ed926-d5b6-7c2d-14e5-1060ccf51fdc"
2024-05-07T18:36:08.103Z [WARN]  server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to contact: server-id=server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc time=50.204137ms
2024-05-07T18:36:08.103Z [WARN]  server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to contact quorum of nodes, stepping down
2024-05-07T18:36:08.103Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: entering follower state: follower="Node at a15824cc-2f30-201a-6047-b1755b6ab586 [Follower]" leader-address= leader-id=
2024-05-07T18:36:08.103Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:08.105Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to heartbeat to: peer=344ed926-d5b6-7c2d-14e5-1060ccf51fdc backoff time=20ms error="failed to connect to peer: 344ed926-d5b6-7c2d-14e5-1060ccf51fdc"
2024-05-07T18:36:08.109Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to appendEntries to: peer="{Voter server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc 344ed926-d5b6-7c2d-14e5-1060ccf51fdc}" error="failed to connect to peer: 344ed926-d5b6-7c2d-14e5-1060ccf51fdc"
2024-05-07T18:36:08.151Z [WARN]  server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: heartbeat timeout reached, starting election: last-leader-addr=a15824cc-2f30-201a-6047-b1755b6ab586 last-leader-id=server-a15824cc-2f30-201a-6047-b1755b6ab586
2024-05-07T18:36:08.151Z [INFO]  server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: entering candidate state: node="Node at 344ed926-d5b6-7c2d-14e5-1060ccf51fdc [Candidate]" term=3
2024-05-07T18:36:08.151Z [ERROR] server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: failed to make requestVote RPC: target="{Voter server-a15824cc-2f30-201a-6047-b1755b6ab586 a15824cc-2f30-201a-6047-b1755b6ab586}" error="failed to connect to peer: a15824cc-2f30-201a-6047-b1755b6ab586" term=3
2024-05-07T18:36:08.167Z [WARN]  server-a15824cc-2f30-201a-6047-b1755b6ab586: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:08.167Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: entering candidate state: node="Node at a15824cc-2f30-201a-6047-b1755b6ab586 [Candidate]" term=3
2024-05-07T18:36:08.167Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to make requestVote RPC: target="{Voter server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc 344ed926-d5b6-7c2d-14e5-1060ccf51fdc}" error="failed to connect to peer: 344ed926-d5b6-7c2d-14e5-1060ccf51fdc" term=3
2024-05-07T18:36:08.230Z [WARN]  server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: Election timeout reached, restarting election
2024-05-07T18:36:08.230Z [INFO]  server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: entering candidate state: node="Node at 344ed926-d5b6-7c2d-14e5-1060ccf51fdc [Candidate]" term=4
2024-05-07T18:36:08.230Z [ERROR] server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: failed to make requestVote RPC: target="{Voter server-a15824cc-2f30-201a-6047-b1755b6ab586 a15824cc-2f30-201a-6047-b1755b6ab586}" error="failed to connect to peer: a15824cc-2f30-201a-6047-b1755b6ab586" term=4
2024-05-07T18:36:08.256Z [WARN]  server-a15824cc-2f30-201a-6047-b1755b6ab586: Election timeout reached, restarting election
2024-05-07T18:36:08.256Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: entering candidate state: node="Node at a15824cc-2f30-201a-6047-b1755b6ab586 [Candidate]" term=4
2024-05-07T18:36:08.256Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to make requestVote RPC: target="{Voter server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc 344ed926-d5b6-7c2d-14e5-1060ccf51fdc}" error="failed to connect to peer: 344ed926-d5b6-7c2d-14e5-1060ccf51fdc" term=4
2024-05-07T18:36:08.273Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:08.167986463 +0000 UTC m=+105.769104713 (0 nodes), 64.155886ms from start of poll, 392.179392ms from cluster start. Timeout at 2024-05-07 18:36:08.273746641 +0000 UTC m=+105.874864901, 105.760188ms after stability
2024-05-07T18:36:08.320Z [WARN]  server-a15824cc-2f30-201a-6047-b1755b6ab586: Election timeout reached, restarting election
2024-05-07T18:36:08.320Z [INFO]  server-a15824cc-2f30-201a-6047-b1755b6ab586: entering candidate state: node="Node at a15824cc-2f30-201a-6047-b1755b6ab586 [Candidate]" term=5
2024-05-07T18:36:08.320Z [ERROR] server-a15824cc-2f30-201a-6047-b1755b6ab586: failed to make requestVote RPC: target="{Voter server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc 344ed926-d5b6-7c2d-14e5-1060ccf51fdc}" error="failed to connect to peer: 344ed926-d5b6-7c2d-14e5-1060ccf51fdc" term=5
2024-05-07T18:36:08.327Z [WARN]  server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: Election timeout reached, restarting election
2024-05-07T18:36:08.327Z [INFO]  server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: entering candidate state: node="Node at 344ed926-d5b6-7c2d-14e5-1060ccf51fdc [Candidate]" term=5
2024-05-07T18:36:08.327Z [ERROR] server-344ed926-d5b6-7c2d-14e5-1060ccf51fdc: failed to make requestVote RPC: target="{Voter server-a15824cc-2f30-201a-6047-b1755b6ab586 a15824cc-2f30-201a-6047-b1755b6ab586}" error="failed to connect to peer: a15824cc-2f30-201a-6047-b1755b6ab586" term=5
--- PASS: TestRaft_LeaderLeaseExpire (0.60s)
=== RUN   TestRaft_Barrier
2024-05-07T18:36:08.380Z [INFO]  server-0d1ec30c-f232-2e5f-b7bc-346628af5211: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d1ec30c-f232-2e5f-b7bc-346628af5211 Address:0d1ec30c-f232-2e5f-b7bc-346628af5211} {Suffrage:Voter ID:server-b49ed41b-12eb-74b0-f690-569449250a9d Address:b49ed41b-12eb-74b0-f690-569449250a9d} {Suffrage:Voter ID:server-919f58a0-f644-2f5c-41c6-99f9ea2443fd Address:919f58a0-f644-2f5c-41c6-99f9ea2443fd}]"
2024-05-07T18:36:08.380Z [INFO]  server-0d1ec30c-f232-2e5f-b7bc-346628af5211: entering follower state: follower="Node at 0d1ec30c-f232-2e5f-b7bc-346628af5211 [Follower]" leader-address= leader-id=
2024-05-07T18:36:08.380Z [INFO]  server-b49ed41b-12eb-74b0-f690-569449250a9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d1ec30c-f232-2e5f-b7bc-346628af5211 Address:0d1ec30c-f232-2e5f-b7bc-346628af5211} {Suffrage:Voter ID:server-b49ed41b-12eb-74b0-f690-569449250a9d Address:b49ed41b-12eb-74b0-f690-569449250a9d} {Suffrage:Voter ID:server-919f58a0-f644-2f5c-41c6-99f9ea2443fd Address:919f58a0-f644-2f5c-41c6-99f9ea2443fd}]"
2024-05-07T18:36:08.380Z [INFO]  server-b49ed41b-12eb-74b0-f690-569449250a9d: entering follower state: follower="Node at b49ed41b-12eb-74b0-f690-569449250a9d [Follower]" leader-address= leader-id=
2024-05-07T18:36:08.381Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d1ec30c-f232-2e5f-b7bc-346628af5211 Address:0d1ec30c-f232-2e5f-b7bc-346628af5211} {Suffrage:Voter ID:server-b49ed41b-12eb-74b0-f690-569449250a9d Address:b49ed41b-12eb-74b0-f690-569449250a9d} {Suffrage:Voter ID:server-919f58a0-f644-2f5c-41c6-99f9ea2443fd Address:919f58a0-f644-2f5c-41c6-99f9ea2443fd}]"
2024-05-07T18:36:08.381Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:08.381Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: entering follower state: follower="Node at 919f58a0-f644-2f5c-41c6-99f9ea2443fd [Follower]" leader-address= leader-id=
2024-05-07T18:36:08.439Z [WARN]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:08.439Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: entering candidate state: node="Node at 919f58a0-f644-2f5c-41c6-99f9ea2443fd [Candidate]" term=2
2024-05-07T18:36:08.439Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: election won: term=2 tally=2
2024-05-07T18:36:08.439Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: entering leader state: leader="Node at 919f58a0-f644-2f5c-41c6-99f9ea2443fd [Leader]"
2024-05-07T18:36:08.439Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: added peer, starting replication: peer=server-0d1ec30c-f232-2e5f-b7bc-346628af5211
2024-05-07T18:36:08.439Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: added peer, starting replication: peer=server-b49ed41b-12eb-74b0-f690-569449250a9d
2024-05-07T18:36:08.439Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: pipelining replication: peer="{Voter server-b49ed41b-12eb-74b0-f690-569449250a9d b49ed41b-12eb-74b0-f690-569449250a9d}"
2024-05-07T18:36:08.439Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: pipelining replication: peer="{Voter server-0d1ec30c-f232-2e5f-b7bc-346628af5211 0d1ec30c-f232-2e5f-b7bc-346628af5211}"
2024-05-07T18:36:08.545Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:08.439701127 +0000 UTC m=+106.040819367 (1 nodes), 58.59266ms from start of poll, 58.922593ms from cluster start. Timeout at 2024-05-07 18:36:08.545808641 +0000 UTC m=+106.146926891, 106.107524ms after stability
2024-05-07T18:36:08.566Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: aborting pipeline replication: peer="{Voter server-b49ed41b-12eb-74b0-f690-569449250a9d b49ed41b-12eb-74b0-f690-569449250a9d}"
2024-05-07T18:36:08.566Z [INFO]  server-919f58a0-f644-2f5c-41c6-99f9ea2443fd: aborting pipeline replication: peer="{Voter server-0d1ec30c-f232-2e5f-b7bc-346628af5211 0d1ec30c-f232-2e5f-b7bc-346628af5211}"
--- PASS: TestRaft_Barrier (0.19s)
=== RUN   TestRaft_VerifyLeader
2024-05-07T18:36:08.567Z [INFO]  server-c68a3532-f728-8e5d-b771-b3af7a0e3d8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c68a3532-f728-8e5d-b771-b3af7a0e3d8a Address:c68a3532-f728-8e5d-b771-b3af7a0e3d8a} {Suffrage:Voter ID:server-e8e6b981-fb8e-d060-aa63-5d6d221350f6 Address:e8e6b981-fb8e-d060-aa63-5d6d221350f6} {Suffrage:Voter ID:server-6b1232a0-5861-de70-2ba3-dd1949262ab8 Address:6b1232a0-5861-de70-2ba3-dd1949262ab8}]"
2024-05-07T18:36:08.567Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c68a3532-f728-8e5d-b771-b3af7a0e3d8a Address:c68a3532-f728-8e5d-b771-b3af7a0e3d8a} {Suffrage:Voter ID:server-e8e6b981-fb8e-d060-aa63-5d6d221350f6 Address:e8e6b981-fb8e-d060-aa63-5d6d221350f6} {Suffrage:Voter ID:server-6b1232a0-5861-de70-2ba3-dd1949262ab8 Address:6b1232a0-5861-de70-2ba3-dd1949262ab8}]"
2024-05-07T18:36:08.567Z [INFO]  server-c68a3532-f728-8e5d-b771-b3af7a0e3d8a: entering follower state: follower="Node at c68a3532-f728-8e5d-b771-b3af7a0e3d8a [Follower]" leader-address= leader-id=
2024-05-07T18:36:08.567Z [INFO]  server-6b1232a0-5861-de70-2ba3-dd1949262ab8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c68a3532-f728-8e5d-b771-b3af7a0e3d8a Address:c68a3532-f728-8e5d-b771-b3af7a0e3d8a} {Suffrage:Voter ID:server-e8e6b981-fb8e-d060-aa63-5d6d221350f6 Address:e8e6b981-fb8e-d060-aa63-5d6d221350f6} {Suffrage:Voter ID:server-6b1232a0-5861-de70-2ba3-dd1949262ab8 Address:6b1232a0-5861-de70-2ba3-dd1949262ab8}]"
2024-05-07T18:36:08.567Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:08.567Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: entering follower state: follower="Node at e8e6b981-fb8e-d060-aa63-5d6d221350f6 [Follower]" leader-address= leader-id=
2024-05-07T18:36:08.567Z [INFO]  server-6b1232a0-5861-de70-2ba3-dd1949262ab8: entering follower state: follower="Node at 6b1232a0-5861-de70-2ba3-dd1949262ab8 [Follower]" leader-address= leader-id=
2024-05-07T18:36:08.636Z [WARN]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:08.636Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: entering candidate state: node="Node at e8e6b981-fb8e-d060-aa63-5d6d221350f6 [Candidate]" term=2
2024-05-07T18:36:08.636Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: election won: term=2 tally=2
2024-05-07T18:36:08.636Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: entering leader state: leader="Node at e8e6b981-fb8e-d060-aa63-5d6d221350f6 [Leader]"
2024-05-07T18:36:08.636Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: added peer, starting replication: peer=server-c68a3532-f728-8e5d-b771-b3af7a0e3d8a
2024-05-07T18:36:08.636Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: added peer, starting replication: peer=server-6b1232a0-5861-de70-2ba3-dd1949262ab8
2024-05-07T18:36:08.636Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: pipelining replication: peer="{Voter server-6b1232a0-5861-de70-2ba3-dd1949262ab8 6b1232a0-5861-de70-2ba3-dd1949262ab8}"
2024-05-07T18:36:08.636Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: pipelining replication: peer="{Voter server-c68a3532-f728-8e5d-b771-b3af7a0e3d8a c68a3532-f728-8e5d-b771-b3af7a0e3d8a}"
2024-05-07T18:36:08.741Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:08.6366927 +0000 UTC m=+106.237810950 (1 nodes), 69.277522ms from start of poll, 69.547965ms from cluster start. Timeout at 2024-05-07 18:36:08.741921059 +0000 UTC m=+106.343039309, 105.228359ms after stability
2024-05-07T18:36:08.742Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: aborting pipeline replication: peer="{Voter server-6b1232a0-5861-de70-2ba3-dd1949262ab8 6b1232a0-5861-de70-2ba3-dd1949262ab8}"
2024-05-07T18:36:08.742Z [INFO]  server-e8e6b981-fb8e-d060-aa63-5d6d221350f6: aborting pipeline replication: peer="{Voter server-c68a3532-f728-8e5d-b771-b3af7a0e3d8a c68a3532-f728-8e5d-b771-b3af7a0e3d8a}"
--- PASS: TestRaft_VerifyLeader (0.18s)
=== RUN   TestRaft_VerifyLeader_Single
2024-05-07T18:36:08.742Z [INFO]  server-cdd1d4e1-ecd2-39a4-d907-6841c23cb033: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd1d4e1-ecd2-39a4-d907-6841c23cb033 Address:cdd1d4e1-ecd2-39a4-d907-6841c23cb033}]"
2024-05-07T18:36:08.742Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:08.742Z [INFO]  server-cdd1d4e1-ecd2-39a4-d907-6841c23cb033: entering follower state: follower="Node at cdd1d4e1-ecd2-39a4-d907-6841c23cb033 [Follower]" leader-address= leader-id=
2024-05-07T18:36:08.797Z [WARN]  server-cdd1d4e1-ecd2-39a4-d907-6841c23cb033: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:08.797Z [INFO]  server-cdd1d4e1-ecd2-39a4-d907-6841c23cb033: entering candidate state: node="Node at cdd1d4e1-ecd2-39a4-d907-6841c23cb033 [Candidate]" term=2
2024-05-07T18:36:08.797Z [INFO]  server-cdd1d4e1-ecd2-39a4-d907-6841c23cb033: election won: term=2 tally=1
2024-05-07T18:36:08.797Z [INFO]  server-cdd1d4e1-ecd2-39a4-d907-6841c23cb033: entering leader state: leader="Node at cdd1d4e1-ecd2-39a4-d907-6841c23cb033 [Leader]"
2024-05-07T18:36:08.903Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:08.797430633 +0000 UTC m=+106.398548883 (1 nodes), 54.672608ms from start of poll, 54.808821ms from cluster start. Timeout at 2024-05-07 18:36:08.903409358 +0000 UTC m=+106.504527608, 105.978725ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.17s)
=== RUN   TestRaft_VerifyLeader_Fail
2024-05-07T18:36:08.910Z [INFO]  server-598248b0-6adf-e74b-7062-43db3112c055: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-598248b0-6adf-e74b-7062-43db3112c055 Address:598248b0-6adf-e74b-7062-43db3112c055} {Suffrage:Voter ID:server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879 Address:dae19eb8-f8ce-a4aa-d6c4-57e178fad879}]"
2024-05-07T18:36:08.910Z [INFO]  server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-598248b0-6adf-e74b-7062-43db3112c055 Address:598248b0-6adf-e74b-7062-43db3112c055} {Suffrage:Voter ID:server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879 Address:dae19eb8-f8ce-a4aa-d6c4-57e178fad879}]"
2024-05-07T18:36:08.910Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:08.910Z [INFO]  server-598248b0-6adf-e74b-7062-43db3112c055: entering follower state: follower="Node at 598248b0-6adf-e74b-7062-43db3112c055 [Follower]" leader-address= leader-id=
2024-05-07T18:36:08.910Z [INFO]  server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: entering follower state: follower="Node at dae19eb8-f8ce-a4aa-d6c4-57e178fad879 [Follower]" leader-address= leader-id=
2024-05-07T18:36:08.973Z [WARN]  server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:08.973Z [INFO]  server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: entering candidate state: node="Node at dae19eb8-f8ce-a4aa-d6c4-57e178fad879 [Candidate]" term=2
2024-05-07T18:36:08.973Z [INFO]  server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: election won: term=2 tally=2
2024-05-07T18:36:08.973Z [INFO]  server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: entering leader state: leader="Node at dae19eb8-f8ce-a4aa-d6c4-57e178fad879 [Leader]"
2024-05-07T18:36:08.973Z [INFO]  server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: added peer, starting replication: peer=server-598248b0-6adf-e74b-7062-43db3112c055
2024-05-07T18:36:08.973Z [INFO]  server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: pipelining replication: peer="{Voter server-598248b0-6adf-e74b-7062-43db3112c055 598248b0-6adf-e74b-7062-43db3112c055}"
2024-05-07T18:36:09.079Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:08.973610669 +0000 UTC m=+106.574728919 (1 nodes), 63.00138ms from start of poll, 63.212412ms from cluster start. Timeout at 2024-05-07 18:36:09.079150286 +0000 UTC m=+106.680268536, 105.539617ms after stability
2024-05-07T18:36:09.079Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:09.184Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:09.07925469 +0000 UTC m=+106.680372980 (1 nodes), 1.652µs from start of poll, 168.856473ms from cluster start. Timeout at 2024-05-07 18:36:09.184355382 +0000 UTC m=+106.785473632, 105.100652ms after stability
2024-05-07T18:36:09.189Z [ERROR] server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: peer has newer term, stopping replication: peer="{Voter server-598248b0-6adf-e74b-7062-43db3112c055 598248b0-6adf-e74b-7062-43db3112c055}"
2024-05-07T18:36:09.189Z [INFO]  server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: aborting pipeline replication: peer="{Voter server-598248b0-6adf-e74b-7062-43db3112c055 598248b0-6adf-e74b-7062-43db3112c055}"
2024-05-07T18:36:09.189Z [INFO]  server-dae19eb8-f8ce-a4aa-d6c4-57e178fad879: entering follower state: follower="Node at dae19eb8-f8ce-a4aa-d6c4-57e178fad879 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.28s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2024-05-07T18:36:09.193Z [INFO]  server-09e35270-5624-2f88-dbfa-96f391c5c5fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09e35270-5624-2f88-dbfa-96f391c5c5fc Address:09e35270-5624-2f88-dbfa-96f391c5c5fc} {Suffrage:Voter ID:server-3f226930-f524-2e9c-51b6-b55574231d4e Address:3f226930-f524-2e9c-51b6-b55574231d4e} {Suffrage:Voter ID:server-8860b582-adf3-0e9a-d8c4-d5186777f9c6 Address:8860b582-adf3-0e9a-d8c4-d5186777f9c6}]"
2024-05-07T18:36:09.193Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09e35270-5624-2f88-dbfa-96f391c5c5fc Address:09e35270-5624-2f88-dbfa-96f391c5c5fc} {Suffrage:Voter ID:server-3f226930-f524-2e9c-51b6-b55574231d4e Address:3f226930-f524-2e9c-51b6-b55574231d4e} {Suffrage:Voter ID:server-8860b582-adf3-0e9a-d8c4-d5186777f9c6 Address:8860b582-adf3-0e9a-d8c4-d5186777f9c6}]"
2024-05-07T18:36:09.193Z [INFO]  server-8860b582-adf3-0e9a-d8c4-d5186777f9c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09e35270-5624-2f88-dbfa-96f391c5c5fc Address:09e35270-5624-2f88-dbfa-96f391c5c5fc} {Suffrage:Voter ID:server-3f226930-f524-2e9c-51b6-b55574231d4e Address:3f226930-f524-2e9c-51b6-b55574231d4e} {Suffrage:Voter ID:server-8860b582-adf3-0e9a-d8c4-d5186777f9c6 Address:8860b582-adf3-0e9a-d8c4-d5186777f9c6}]"
2024-05-07T18:36:09.193Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:09.193Z [INFO]  server-09e35270-5624-2f88-dbfa-96f391c5c5fc: entering follower state: follower="Node at 09e35270-5624-2f88-dbfa-96f391c5c5fc [Follower]" leader-address= leader-id=
2024-05-07T18:36:09.193Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: entering follower state: follower="Node at 3f226930-f524-2e9c-51b6-b55574231d4e [Follower]" leader-address= leader-id=
2024-05-07T18:36:09.193Z [INFO]  server-8860b582-adf3-0e9a-d8c4-d5186777f9c6: entering follower state: follower="Node at 8860b582-adf3-0e9a-d8c4-d5186777f9c6 [Follower]" leader-address= leader-id=
2024-05-07T18:36:09.244Z [WARN]  server-3f226930-f524-2e9c-51b6-b55574231d4e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:09.245Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: entering candidate state: node="Node at 3f226930-f524-2e9c-51b6-b55574231d4e [Candidate]" term=2
2024-05-07T18:36:09.245Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: election won: term=2 tally=2
2024-05-07T18:36:09.245Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: entering leader state: leader="Node at 3f226930-f524-2e9c-51b6-b55574231d4e [Leader]"
2024-05-07T18:36:09.245Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: added peer, starting replication: peer=server-09e35270-5624-2f88-dbfa-96f391c5c5fc
2024-05-07T18:36:09.245Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: added peer, starting replication: peer=server-8860b582-adf3-0e9a-d8c4-d5186777f9c6
2024-05-07T18:36:09.245Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: pipelining replication: peer="{Voter server-8860b582-adf3-0e9a-d8c4-d5186777f9c6 8860b582-adf3-0e9a-d8c4-d5186777f9c6}"
2024-05-07T18:36:09.245Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: pipelining replication: peer="{Voter server-09e35270-5624-2f88-dbfa-96f391c5c5fc 09e35270-5624-2f88-dbfa-96f391c5c5fc}"
2024-05-07T18:36:09.350Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:09.245131361 +0000 UTC m=+106.846249611 (1 nodes), 51.420518ms from start of poll, 51.634968ms from cluster start. Timeout at 2024-05-07 18:36:09.350552439 +0000 UTC m=+106.951670699, 105.421088ms after stability
2024-05-07T18:36:09.356Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:09.461Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:09.356580841 +0000 UTC m=+106.957699101 (2 nodes), 1.863µs from start of poll, 163.084458ms from cluster start. Timeout at 2024-05-07 18:36:09.461673428 +0000 UTC m=+107.062791678, 105.092577ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at 09e35270-5624-2f88-dbfa-96f391c5c5fc [Follower]
2024-05-07T18:36:09.461Z [ERROR] server-3f226930-f524-2e9c-51b6-b55574231d4e: failed to heartbeat to: peer=09e35270-5624-2f88-dbfa-96f391c5c5fc backoff time=10ms error="failed to connect to peer: 09e35270-5624-2f88-dbfa-96f391c5c5fc"
2024-05-07T18:36:09.461Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: aborting pipeline replication: peer="{Voter server-09e35270-5624-2f88-dbfa-96f391c5c5fc 09e35270-5624-2f88-dbfa-96f391c5c5fc}"
2024-05-07T18:36:09.461Z [INFO]  server-3f226930-f524-2e9c-51b6-b55574231d4e: aborting pipeline replication: peer="{Voter server-8860b582-adf3-0e9a-d8c4-d5186777f9c6 8860b582-adf3-0e9a-d8c4-d5186777f9c6}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.27s)
=== RUN   TestRaft_NotifyCh
2024-05-07T18:36:09.462Z [INFO]  server-435fbfc1-ad33-ab48-3f41-b73f52db7fb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-435fbfc1-ad33-ab48-3f41-b73f52db7fb9 Address:435fbfc1-ad33-ab48-3f41-b73f52db7fb9}]"
2024-05-07T18:36:09.462Z [INFO]  server-435fbfc1-ad33-ab48-3f41-b73f52db7fb9: entering follower state: follower="Node at 435fbfc1-ad33-ab48-3f41-b73f52db7fb9 [Follower]" leader-address= leader-id=
2024-05-07T18:36:09.554Z [WARN]  server-435fbfc1-ad33-ab48-3f41-b73f52db7fb9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:09.554Z [INFO]  server-435fbfc1-ad33-ab48-3f41-b73f52db7fb9: entering candidate state: node="Node at 435fbfc1-ad33-ab48-3f41-b73f52db7fb9 [Candidate]" term=2
2024-05-07T18:36:09.554Z [INFO]  server-435fbfc1-ad33-ab48-3f41-b73f52db7fb9: election won: term=2 tally=1
2024-05-07T18:36:09.554Z [INFO]  server-435fbfc1-ad33-ab48-3f41-b73f52db7fb9: entering leader state: leader="Node at 435fbfc1-ad33-ab48-3f41-b73f52db7fb9 [Leader]"
--- PASS: TestRaft_NotifyCh (0.09s)
=== RUN   TestRaft_AppendEntry
2024-05-07T18:36:09.554Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-837ab236-c5f0-a01e-9a2d-73f23196b85d Address:837ab236-c5f0-a01e-9a2d-73f23196b85d} {Suffrage:Voter ID:server-a1bbbb3c-8c29-1db7-1248-c101600e6c36 Address:a1bbbb3c-8c29-1db7-1248-c101600e6c36} {Suffrage:Voter ID:server-79844337-7d89-5012-bd9d-c2359d05d9cd Address:79844337-7d89-5012-bd9d-c2359d05d9cd}]"
2024-05-07T18:36:09.554Z [INFO]  server-a1bbbb3c-8c29-1db7-1248-c101600e6c36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-837ab236-c5f0-a01e-9a2d-73f23196b85d Address:837ab236-c5f0-a01e-9a2d-73f23196b85d} {Suffrage:Voter ID:server-a1bbbb3c-8c29-1db7-1248-c101600e6c36 Address:a1bbbb3c-8c29-1db7-1248-c101600e6c36} {Suffrage:Voter ID:server-79844337-7d89-5012-bd9d-c2359d05d9cd Address:79844337-7d89-5012-bd9d-c2359d05d9cd}]"
2024-05-07T18:36:09.555Z [INFO]  server-79844337-7d89-5012-bd9d-c2359d05d9cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-837ab236-c5f0-a01e-9a2d-73f23196b85d Address:837ab236-c5f0-a01e-9a2d-73f23196b85d} {Suffrage:Voter ID:server-a1bbbb3c-8c29-1db7-1248-c101600e6c36 Address:a1bbbb3c-8c29-1db7-1248-c101600e6c36} {Suffrage:Voter ID:server-79844337-7d89-5012-bd9d-c2359d05d9cd Address:79844337-7d89-5012-bd9d-c2359d05d9cd}]"
2024-05-07T18:36:09.555Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:09.555Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: entering follower state: follower="Node at 837ab236-c5f0-a01e-9a2d-73f23196b85d [Follower]" leader-address= leader-id=
2024-05-07T18:36:09.555Z [INFO]  server-a1bbbb3c-8c29-1db7-1248-c101600e6c36: entering follower state: follower="Node at a1bbbb3c-8c29-1db7-1248-c101600e6c36 [Follower]" leader-address= leader-id=
2024-05-07T18:36:09.555Z [INFO]  server-79844337-7d89-5012-bd9d-c2359d05d9cd: entering follower state: follower="Node at 79844337-7d89-5012-bd9d-c2359d05d9cd [Follower]" leader-address= leader-id=
2024-05-07T18:36:09.610Z [WARN]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:09.610Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: entering candidate state: node="Node at 837ab236-c5f0-a01e-9a2d-73f23196b85d [Candidate]" term=2
2024-05-07T18:36:09.610Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: election won: term=2 tally=2
2024-05-07T18:36:09.610Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: entering leader state: leader="Node at 837ab236-c5f0-a01e-9a2d-73f23196b85d [Leader]"
2024-05-07T18:36:09.610Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: added peer, starting replication: peer=server-a1bbbb3c-8c29-1db7-1248-c101600e6c36
2024-05-07T18:36:09.610Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: added peer, starting replication: peer=server-79844337-7d89-5012-bd9d-c2359d05d9cd
2024-05-07T18:36:09.610Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: pipelining replication: peer="{Voter server-79844337-7d89-5012-bd9d-c2359d05d9cd 79844337-7d89-5012-bd9d-c2359d05d9cd}"
2024-05-07T18:36:09.610Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: pipelining replication: peer="{Voter server-a1bbbb3c-8c29-1db7-1248-c101600e6c36 a1bbbb3c-8c29-1db7-1248-c101600e6c36}"
2024-05-07T18:36:09.716Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:09.610386125 +0000 UTC m=+107.211504375 (2 nodes), 55.332975ms from start of poll, 55.542055ms from cluster start. Timeout at 2024-05-07 18:36:09.716323042 +0000 UTC m=+107.317441292, 105.936917ms after stability
2024-05-07T18:36:09.716Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:09.821Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:09.71639183 +0000 UTC m=+107.317510080 (1 nodes), 842ns from start of poll, 161.54776ms from cluster start. Timeout at 2024-05-07 18:36:09.821610421 +0000 UTC m=+107.422728671, 105.218591ms after stability
2024-05-07T18:36:09.821Z [WARN]  server-a1bbbb3c-8c29-1db7-1248-c101600e6c36: clearing log suffix: from=1 to=2
2024-05-07T18:36:09.821Z [WARN]  server-a1bbbb3c-8c29-1db7-1248-c101600e6c36: skipping application of old log: index=1
2024-05-07T18:36:09.821Z [WARN]  server-a1bbbb3c-8c29-1db7-1248-c101600e6c36: skipping application of old log: index=1
2024-05-07T18:36:09.821Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: aborting pipeline replication: peer="{Voter server-79844337-7d89-5012-bd9d-c2359d05d9cd 79844337-7d89-5012-bd9d-c2359d05d9cd}"
2024-05-07T18:36:09.821Z [INFO]  server-837ab236-c5f0-a01e-9a2d-73f23196b85d: aborting pipeline replication: peer="{Voter server-a1bbbb3c-8c29-1db7-1248-c101600e6c36 a1bbbb3c-8c29-1db7-1248-c101600e6c36}"
--- PASS: TestRaft_AppendEntry (0.27s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2024-05-07T18:36:09.827Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90 Address:f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90} {Suffrage:Voter ID:server-08c12f97-2503-37de-8746-fc298ae8aeb9 Address:08c12f97-2503-37de-8746-fc298ae8aeb9} {Suffrage:Voter ID:server-d05a0a81-061d-22fc-068d-b5850b262db8 Address:d05a0a81-061d-22fc-068d-b5850b262db8}]"
2024-05-07T18:36:09.827Z [INFO]  server-08c12f97-2503-37de-8746-fc298ae8aeb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90 Address:f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90} {Suffrage:Voter ID:server-08c12f97-2503-37de-8746-fc298ae8aeb9 Address:08c12f97-2503-37de-8746-fc298ae8aeb9} {Suffrage:Voter ID:server-d05a0a81-061d-22fc-068d-b5850b262db8 Address:d05a0a81-061d-22fc-068d-b5850b262db8}]"
2024-05-07T18:36:09.827Z [INFO]  server-d05a0a81-061d-22fc-068d-b5850b262db8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90 Address:f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90} {Suffrage:Voter ID:server-08c12f97-2503-37de-8746-fc298ae8aeb9 Address:08c12f97-2503-37de-8746-fc298ae8aeb9} {Suffrage:Voter ID:server-d05a0a81-061d-22fc-068d-b5850b262db8 Address:d05a0a81-061d-22fc-068d-b5850b262db8}]"
2024-05-07T18:36:09.827Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:09.827Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: entering follower state: follower="Node at f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90 [Follower]" leader-address= leader-id=
2024-05-07T18:36:09.827Z [INFO]  server-08c12f97-2503-37de-8746-fc298ae8aeb9: entering follower state: follower="Node at 08c12f97-2503-37de-8746-fc298ae8aeb9 [Follower]" leader-address= leader-id=
2024-05-07T18:36:09.827Z [INFO]  server-d05a0a81-061d-22fc-068d-b5850b262db8: entering follower state: follower="Node at d05a0a81-061d-22fc-068d-b5850b262db8 [Follower]" leader-address= leader-id=
2024-05-07T18:36:09.882Z [WARN]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:09.882Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: entering candidate state: node="Node at f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90 [Candidate]" term=2
2024-05-07T18:36:09.882Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: election won: term=2 tally=2
2024-05-07T18:36:09.882Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: entering leader state: leader="Node at f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90 [Leader]"
2024-05-07T18:36:09.882Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: added peer, starting replication: peer=server-08c12f97-2503-37de-8746-fc298ae8aeb9
2024-05-07T18:36:09.882Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: added peer, starting replication: peer=server-d05a0a81-061d-22fc-068d-b5850b262db8
2024-05-07T18:36:09.882Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: pipelining replication: peer="{Voter server-08c12f97-2503-37de-8746-fc298ae8aeb9 08c12f97-2503-37de-8746-fc298ae8aeb9}"
2024-05-07T18:36:09.882Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: pipelining replication: peer="{Voter server-d05a0a81-061d-22fc-068d-b5850b262db8 d05a0a81-061d-22fc-068d-b5850b262db8}"
2024-05-07T18:36:09.988Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:09.882598665 +0000 UTC m=+107.483716915 (2 nodes), 55.371147ms from start of poll, 55.60939ms from cluster start. Timeout at 2024-05-07 18:36:09.988205999 +0000 UTC m=+107.589324249, 105.607334ms after stability
2024-05-07T18:36:09.988Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:10.094Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:09.988262424 +0000 UTC m=+107.589380674 (1 nodes), 651ns from start of poll, 161.273149ms from cluster start. Timeout at 2024-05-07 18:36:10.094351875 +0000 UTC m=+107.695470135, 106.089461ms after stability
2024-05-07T18:36:10.094Z [WARN]  server-d05a0a81-061d-22fc-068d-b5850b262db8: rejecting vote request since we have a leader: from=08c12f97-2503-37de-8746-fc298ae8aeb9 leader=f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90 leader-id=server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90
2024-05-07T18:36:10.094Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: aborting pipeline replication: peer="{Voter server-d05a0a81-061d-22fc-068d-b5850b262db8 d05a0a81-061d-22fc-068d-b5850b262db8}"
2024-05-07T18:36:10.094Z [INFO]  server-f0c23cc0-2d8d-ee4e-4eb4-a64fda770c90: aborting pipeline replication: peer="{Voter server-08c12f97-2503-37de-8746-fc298ae8aeb9 08c12f97-2503-37de-8746-fc298ae8aeb9}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.27s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2024-05-07T18:36:10.097Z [INFO]  server-59232f3b-2df3-2f2b-fcd9-60734f2b34b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59232f3b-2df3-2f2b-fcd9-60734f2b34b3 Address:59232f3b-2df3-2f2b-fcd9-60734f2b34b3} {Suffrage:Voter ID:server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec Address:de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec} {Suffrage:Voter ID:server-9776e34d-a197-7ab5-e7e6-32ed1ee60766 Address:9776e34d-a197-7ab5-e7e6-32ed1ee60766}]"
2024-05-07T18:36:10.097Z [INFO]  server-59232f3b-2df3-2f2b-fcd9-60734f2b34b3: entering follower state: follower="Node at 59232f3b-2df3-2f2b-fcd9-60734f2b34b3 [Follower]" leader-address= leader-id=
2024-05-07T18:36:10.097Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59232f3b-2df3-2f2b-fcd9-60734f2b34b3 Address:59232f3b-2df3-2f2b-fcd9-60734f2b34b3} {Suffrage:Voter ID:server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec Address:de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec} {Suffrage:Voter ID:server-9776e34d-a197-7ab5-e7e6-32ed1ee60766 Address:9776e34d-a197-7ab5-e7e6-32ed1ee60766}]"
2024-05-07T18:36:10.097Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: entering follower state: follower="Node at de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec [Follower]" leader-address= leader-id=
2024-05-07T18:36:10.097Z [INFO]  server-9776e34d-a197-7ab5-e7e6-32ed1ee60766: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59232f3b-2df3-2f2b-fcd9-60734f2b34b3 Address:59232f3b-2df3-2f2b-fcd9-60734f2b34b3} {Suffrage:Voter ID:server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec Address:de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec} {Suffrage:Voter ID:server-9776e34d-a197-7ab5-e7e6-32ed1ee60766 Address:9776e34d-a197-7ab5-e7e6-32ed1ee60766}]"
2024-05-07T18:36:10.097Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:10.097Z [INFO]  server-9776e34d-a197-7ab5-e7e6-32ed1ee60766: entering follower state: follower="Node at 9776e34d-a197-7ab5-e7e6-32ed1ee60766 [Follower]" leader-address= leader-id=
2024-05-07T18:36:10.163Z [WARN]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:10.163Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: entering candidate state: node="Node at de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec [Candidate]" term=2
2024-05-07T18:36:10.163Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: election won: term=2 tally=2
2024-05-07T18:36:10.163Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: entering leader state: leader="Node at de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec [Leader]"
2024-05-07T18:36:10.163Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: added peer, starting replication: peer=server-59232f3b-2df3-2f2b-fcd9-60734f2b34b3
2024-05-07T18:36:10.163Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: added peer, starting replication: peer=server-9776e34d-a197-7ab5-e7e6-32ed1ee60766
2024-05-07T18:36:10.163Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: pipelining replication: peer="{Voter server-9776e34d-a197-7ab5-e7e6-32ed1ee60766 9776e34d-a197-7ab5-e7e6-32ed1ee60766}"
2024-05-07T18:36:10.163Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: pipelining replication: peer="{Voter server-59232f3b-2df3-2f2b-fcd9-60734f2b34b3 59232f3b-2df3-2f2b-fcd9-60734f2b34b3}"
2024-05-07T18:36:10.269Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:10.16365848 +0000 UTC m=+107.764776741 (2 nodes), 66.313399ms from start of poll, 66.758347ms from cluster start. Timeout at 2024-05-07 18:36:10.269242301 +0000 UTC m=+107.870360561, 105.58382ms after stability
2024-05-07T18:36:10.269Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:10.374Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:10.26930623 +0000 UTC m=+107.870424490 (1 nodes), 1.473µs from start of poll, 172.406096ms from cluster start. Timeout at 2024-05-07 18:36:10.37442236 +0000 UTC m=+107.975540620, 105.11613ms after stability
2024-05-07T18:36:10.374Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: aborting pipeline replication: peer="{Voter server-9776e34d-a197-7ab5-e7e6-32ed1ee60766 9776e34d-a197-7ab5-e7e6-32ed1ee60766}"
2024-05-07T18:36:10.374Z [INFO]  server-de8c437c-f9f4-25bc-7a8b-4933a3e1c0ec: aborting pipeline replication: peer="{Voter server-59232f3b-2df3-2f2b-fcd9-60734f2b34b3 59232f3b-2df3-2f2b-fcd9-60734f2b34b3}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.28s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2024-05-07T18:36:10.375Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2820deac-58a5-236c-c024-2ebc7daa5df6 Address:2820deac-58a5-236c-c024-2ebc7daa5df6} {Suffrage:Voter ID:server-22e51bd7-a36b-2995-b119-d0dfa3ec5284 Address:22e51bd7-a36b-2995-b119-d0dfa3ec5284} {Suffrage:Voter ID:server-59581a19-3ebe-c229-6cae-ca625c249872 Address:59581a19-3ebe-c229-6cae-ca625c249872}]"
2024-05-07T18:36:10.375Z [INFO]  server-22e51bd7-a36b-2995-b119-d0dfa3ec5284: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2820deac-58a5-236c-c024-2ebc7daa5df6 Address:2820deac-58a5-236c-c024-2ebc7daa5df6} {Suffrage:Voter ID:server-22e51bd7-a36b-2995-b119-d0dfa3ec5284 Address:22e51bd7-a36b-2995-b119-d0dfa3ec5284} {Suffrage:Voter ID:server-59581a19-3ebe-c229-6cae-ca625c249872 Address:59581a19-3ebe-c229-6cae-ca625c249872}]"
2024-05-07T18:36:10.375Z [INFO]  server-59581a19-3ebe-c229-6cae-ca625c249872: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2820deac-58a5-236c-c024-2ebc7daa5df6 Address:2820deac-58a5-236c-c024-2ebc7daa5df6} {Suffrage:Voter ID:server-22e51bd7-a36b-2995-b119-d0dfa3ec5284 Address:22e51bd7-a36b-2995-b119-d0dfa3ec5284} {Suffrage:Voter ID:server-59581a19-3ebe-c229-6cae-ca625c249872 Address:59581a19-3ebe-c229-6cae-ca625c249872}]"
2024-05-07T18:36:10.375Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:10.375Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: entering follower state: follower="Node at 2820deac-58a5-236c-c024-2ebc7daa5df6 [Follower]" leader-address= leader-id=
2024-05-07T18:36:10.376Z [INFO]  server-22e51bd7-a36b-2995-b119-d0dfa3ec5284: entering follower state: follower="Node at 22e51bd7-a36b-2995-b119-d0dfa3ec5284 [Follower]" leader-address= leader-id=
2024-05-07T18:36:10.376Z [INFO]  server-59581a19-3ebe-c229-6cae-ca625c249872: entering follower state: follower="Node at 59581a19-3ebe-c229-6cae-ca625c249872 [Follower]" leader-address= leader-id=
2024-05-07T18:36:10.429Z [WARN]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:10.429Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: entering candidate state: node="Node at 2820deac-58a5-236c-c024-2ebc7daa5df6 [Candidate]" term=2
2024-05-07T18:36:10.429Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: election won: term=2 tally=2
2024-05-07T18:36:10.429Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: entering leader state: leader="Node at 2820deac-58a5-236c-c024-2ebc7daa5df6 [Leader]"
2024-05-07T18:36:10.429Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: added peer, starting replication: peer=server-22e51bd7-a36b-2995-b119-d0dfa3ec5284
2024-05-07T18:36:10.429Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: added peer, starting replication: peer=server-59581a19-3ebe-c229-6cae-ca625c249872
2024-05-07T18:36:10.429Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: pipelining replication: peer="{Voter server-59581a19-3ebe-c229-6cae-ca625c249872 59581a19-3ebe-c229-6cae-ca625c249872}"
2024-05-07T18:36:10.429Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: pipelining replication: peer="{Voter server-22e51bd7-a36b-2995-b119-d0dfa3ec5284 22e51bd7-a36b-2995-b119-d0dfa3ec5284}"
2024-05-07T18:36:10.534Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:10.429524086 +0000 UTC m=+108.030642336 (1 nodes), 53.572903ms from start of poll, 53.873341ms from cluster start. Timeout at 2024-05-07 18:36:10.534528829 +0000 UTC m=+108.135647079, 105.004743ms after stability
2024-05-07T18:36:10.534Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:10.640Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:10.534600722 +0000 UTC m=+108.135718973 (1 nodes), 792ns from start of poll, 158.949978ms from cluster start. Timeout at 2024-05-07 18:36:10.63999481 +0000 UTC m=+108.241113060, 105.394087ms after stability
2024-05-07T18:36:10.640Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:10.745Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:10.640131123 +0000 UTC m=+108.241249373 (1 nodes), 561ns from start of poll, 264.480378ms from cluster start. Timeout at 2024-05-07 18:36:10.74529893 +0000 UTC m=+108.346417180, 105.167807ms after stability
2024-05-07T18:36:10.745Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:10.850Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:10.745519821 +0000 UTC m=+108.346638071 (1 nodes), 1.323µs from start of poll, 369.869076ms from cluster start. Timeout at 2024-05-07 18:36:10.850773326 +0000 UTC m=+108.451891576, 105.253505ms after stability
2024-05-07T18:36:10.850Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:10.956Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:10.850945226 +0000 UTC m=+108.452063476 (1 nodes), 400ns from start of poll, 475.294481ms from cluster start. Timeout at 2024-05-07 18:36:10.95674074 +0000 UTC m=+108.557858990, 105.795514ms after stability
2024-05-07T18:36:10.956Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:11.062Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:10.956871563 +0000 UTC m=+108.557989813 (1 nodes), 541ns from start of poll, 581.220818ms from cluster start. Timeout at 2024-05-07 18:36:11.062267013 +0000 UTC m=+108.663385263, 105.39545ms after stability
2024-05-07T18:36:11.168Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:11.273Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:11.168117468 +0000 UTC m=+108.769235718 (1 nodes), 1.102µs from start of poll, 792.466723ms from cluster start. Timeout at 2024-05-07 18:36:11.273698483 +0000 UTC m=+108.874816733, 105.581015ms after stability
2024-05-07T18:36:11.273Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: aborting pipeline replication: peer="{Voter server-59581a19-3ebe-c229-6cae-ca625c249872 59581a19-3ebe-c229-6cae-ca625c249872}"
2024-05-07T18:36:11.273Z [INFO]  server-2820deac-58a5-236c-c024-2ebc7daa5df6: aborting pipeline replication: peer="{Voter server-22e51bd7-a36b-2995-b119-d0dfa3ec5284 22e51bd7-a36b-2995-b119-d0dfa3ec5284}"
--- PASS: TestRaft_LeaderID_Propagated (0.96s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2024-05-07T18:36:11.337Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a813af15-9568-4c52-21c5-ae7a43ebc5db Address:a813af15-9568-4c52-21c5-ae7a43ebc5db} {Suffrage:Voter ID:server-1805127a-5509-e32a-5834-f779012ea217 Address:1805127a-5509-e32a-5834-f779012ea217} {Suffrage:Voter ID:server-623d55ee-77b8-e986-90e0-a7546672c29b Address:623d55ee-77b8-e986-90e0-a7546672c29b}]"
2024-05-07T18:36:11.337Z [INFO]  server-1805127a-5509-e32a-5834-f779012ea217: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a813af15-9568-4c52-21c5-ae7a43ebc5db Address:a813af15-9568-4c52-21c5-ae7a43ebc5db} {Suffrage:Voter ID:server-1805127a-5509-e32a-5834-f779012ea217 Address:1805127a-5509-e32a-5834-f779012ea217} {Suffrage:Voter ID:server-623d55ee-77b8-e986-90e0-a7546672c29b Address:623d55ee-77b8-e986-90e0-a7546672c29b}]"
2024-05-07T18:36:11.337Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: entering follower state: follower="Node at a813af15-9568-4c52-21c5-ae7a43ebc5db [Follower]" leader-address= leader-id=
2024-05-07T18:36:11.337Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a813af15-9568-4c52-21c5-ae7a43ebc5db Address:a813af15-9568-4c52-21c5-ae7a43ebc5db} {Suffrage:Voter ID:server-1805127a-5509-e32a-5834-f779012ea217 Address:1805127a-5509-e32a-5834-f779012ea217} {Suffrage:Voter ID:server-623d55ee-77b8-e986-90e0-a7546672c29b Address:623d55ee-77b8-e986-90e0-a7546672c29b}]"
2024-05-07T18:36:11.337Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:11.337Z [INFO]  server-1805127a-5509-e32a-5834-f779012ea217: entering follower state: follower="Node at 1805127a-5509-e32a-5834-f779012ea217 [Follower]" leader-address= leader-id=
2024-05-07T18:36:11.337Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: entering follower state: follower="Node at 623d55ee-77b8-e986-90e0-a7546672c29b [Follower]" leader-address= leader-id=
2024-05-07T18:36:11.393Z [WARN]  server-623d55ee-77b8-e986-90e0-a7546672c29b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:11.393Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: entering candidate state: node="Node at 623d55ee-77b8-e986-90e0-a7546672c29b [Candidate]" term=2
2024-05-07T18:36:11.393Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: election won: term=2 tally=2
2024-05-07T18:36:11.393Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: entering leader state: leader="Node at 623d55ee-77b8-e986-90e0-a7546672c29b [Leader]"
2024-05-07T18:36:11.393Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: added peer, starting replication: peer=server-a813af15-9568-4c52-21c5-ae7a43ebc5db
2024-05-07T18:36:11.393Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: added peer, starting replication: peer=server-1805127a-5509-e32a-5834-f779012ea217
2024-05-07T18:36:11.393Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: pipelining replication: peer="{Voter server-1805127a-5509-e32a-5834-f779012ea217 1805127a-5509-e32a-5834-f779012ea217}"
2024-05-07T18:36:11.393Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: pipelining replication: peer="{Voter server-a813af15-9568-4c52-21c5-ae7a43ebc5db a813af15-9568-4c52-21c5-ae7a43ebc5db}"
2024-05-07T18:36:11.498Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:11.393352984 +0000 UTC m=+108.994471234 (1 nodes), 56.145347ms from start of poll, 56.420098ms from cluster start. Timeout at 2024-05-07 18:36:11.49844537 +0000 UTC m=+109.099563620, 105.092386ms after stability
2024-05-07T18:36:11.498Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:11.604Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:11.498523145 +0000 UTC m=+109.099641395 (1 nodes), 1.002µs from start of poll, 161.590259ms from cluster start. Timeout at 2024-05-07 18:36:11.604049127 +0000 UTC m=+109.205167377, 105.525982ms after stability
2024-05-07T18:36:11.604Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: entering candidate state: node="Node at a813af15-9568-4c52-21c5-ae7a43ebc5db [Candidate]" term=3
2024-05-07T18:36:11.604Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: entering follower state: follower="Node at 623d55ee-77b8-e986-90e0-a7546672c29b [Follower]" leader-address= leader-id=
2024-05-07T18:36:11.604Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:11.604Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: election won: term=3 tally=2
2024-05-07T18:36:11.604Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: entering leader state: leader="Node at a813af15-9568-4c52-21c5-ae7a43ebc5db [Leader]"
2024-05-07T18:36:11.604Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: aborting pipeline replication: peer="{Voter server-a813af15-9568-4c52-21c5-ae7a43ebc5db a813af15-9568-4c52-21c5-ae7a43ebc5db}"
2024-05-07T18:36:11.604Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: added peer, starting replication: peer=server-1805127a-5509-e32a-5834-f779012ea217
2024-05-07T18:36:11.604Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: added peer, starting replication: peer=server-623d55ee-77b8-e986-90e0-a7546672c29b
2024-05-07T18:36:11.604Z [INFO]  server-623d55ee-77b8-e986-90e0-a7546672c29b: aborting pipeline replication: peer="{Voter server-1805127a-5509-e32a-5834-f779012ea217 1805127a-5509-e32a-5834-f779012ea217}"
2024-05-07T18:36:11.604Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: pipelining replication: peer="{Voter server-1805127a-5509-e32a-5834-f779012ea217 1805127a-5509-e32a-5834-f779012ea217}"
2024-05-07T18:36:11.604Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: pipelining replication: peer="{Voter server-623d55ee-77b8-e986-90e0-a7546672c29b 623d55ee-77b8-e986-90e0-a7546672c29b}"
2024-05-07T18:36:11.709Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:11.604225835 +0000 UTC m=+109.205344086 (1 nodes), 14.908µs from start of poll, 267.29295ms from cluster start. Timeout at 2024-05-07 18:36:11.709668303 +0000 UTC m=+109.310786553, 105.442467ms after stability
2024-05-07T18:36:11.709Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: aborting pipeline replication: peer="{Voter server-623d55ee-77b8-e986-90e0-a7546672c29b 623d55ee-77b8-e986-90e0-a7546672c29b}"
2024-05-07T18:36:11.709Z [INFO]  server-a813af15-9568-4c52-21c5-ae7a43ebc5db: aborting pipeline replication: peer="{Voter server-1805127a-5509-e32a-5834-f779012ea217 1805127a-5509-e32a-5834-f779012ea217}"
--- PASS: TestRaft_LeadershipTransfer (0.44s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2024-05-07T18:36:11.778Z [INFO]  server-fa2f8fa8-e19f-eccd-fad7-c85c873a54e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa2f8fa8-e19f-eccd-fad7-c85c873a54e0 Address:fa2f8fa8-e19f-eccd-fad7-c85c873a54e0}]"
2024-05-07T18:36:11.778Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:11.778Z [INFO]  server-fa2f8fa8-e19f-eccd-fad7-c85c873a54e0: entering follower state: follower="Node at fa2f8fa8-e19f-eccd-fad7-c85c873a54e0 [Follower]" leader-address= leader-id=
2024-05-07T18:36:11.841Z [WARN]  server-fa2f8fa8-e19f-eccd-fad7-c85c873a54e0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:11.841Z [INFO]  server-fa2f8fa8-e19f-eccd-fad7-c85c873a54e0: entering candidate state: node="Node at fa2f8fa8-e19f-eccd-fad7-c85c873a54e0 [Candidate]" term=2
2024-05-07T18:36:11.842Z [INFO]  server-fa2f8fa8-e19f-eccd-fad7-c85c873a54e0: election won: term=2 tally=1
2024-05-07T18:36:11.842Z [INFO]  server-fa2f8fa8-e19f-eccd-fad7-c85c873a54e0: entering leader state: leader="Node at fa2f8fa8-e19f-eccd-fad7-c85c873a54e0 [Leader]"
2024-05-07T18:36:11.948Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:11.842053893 +0000 UTC m=+109.443172143 (1 nodes), 63.2437ms from start of poll, 63.43185ms from cluster start. Timeout at 2024-05-07 18:36:11.947961845 +0000 UTC m=+109.549080095, 105.907952ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.28s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2024-05-07T18:36:12.059Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cbec760-f4bf-2a77-6c01-deaffb12a77c Address:2cbec760-f4bf-2a77-6c01-deaffb12a77c} {Suffrage:Voter ID:server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f Address:0e7e1a2b-8d79-b5c3-987f-500ec4aed88f} {Suffrage:Voter ID:server-d3407936-5652-58ee-f84c-42d117f8e408 Address:d3407936-5652-58ee-f84c-42d117f8e408} {Suffrage:Voter ID:server-b36ee052-17b3-fd43-7ff2-3d5138893f00 Address:b36ee052-17b3-fd43-7ff2-3d5138893f00} {Suffrage:Voter ID:server-e6b7053d-26a2-9666-7c6c-317faf67c173 Address:e6b7053d-26a2-9666-7c6c-317faf67c173} {Suffrage:Voter ID:server-2a7a923b-38ae-bd9f-3b81-3062a5006378 Address:2a7a923b-38ae-bd9f-3b81-3062a5006378} {Suffrage:Voter ID:server-7cab0268-da6c-6645-0c22-43dfdbcc39af Address:7cab0268-da6c-6645-0c22-43dfdbcc39af}]"
2024-05-07T18:36:12.059Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: entering follower state: follower="Node at 2cbec760-f4bf-2a77-6c01-deaffb12a77c [Follower]" leader-address= leader-id=
2024-05-07T18:36:12.059Z [INFO]  server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cbec760-f4bf-2a77-6c01-deaffb12a77c Address:2cbec760-f4bf-2a77-6c01-deaffb12a77c} {Suffrage:Voter ID:server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f Address:0e7e1a2b-8d79-b5c3-987f-500ec4aed88f} {Suffrage:Voter ID:server-d3407936-5652-58ee-f84c-42d117f8e408 Address:d3407936-5652-58ee-f84c-42d117f8e408} {Suffrage:Voter ID:server-b36ee052-17b3-fd43-7ff2-3d5138893f00 Address:b36ee052-17b3-fd43-7ff2-3d5138893f00} {Suffrage:Voter ID:server-e6b7053d-26a2-9666-7c6c-317faf67c173 Address:e6b7053d-26a2-9666-7c6c-317faf67c173} {Suffrage:Voter ID:server-2a7a923b-38ae-bd9f-3b81-3062a5006378 Address:2a7a923b-38ae-bd9f-3b81-3062a5006378} {Suffrage:Voter ID:server-7cab0268-da6c-6645-0c22-43dfdbcc39af Address:7cab0268-da6c-6645-0c22-43dfdbcc39af}]"
2024-05-07T18:36:12.060Z [INFO]  server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f: entering follower state: follower="Node at 0e7e1a2b-8d79-b5c3-987f-500ec4aed88f [Follower]" leader-address= leader-id=
2024-05-07T18:36:12.060Z [INFO]  server-d3407936-5652-58ee-f84c-42d117f8e408: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cbec760-f4bf-2a77-6c01-deaffb12a77c Address:2cbec760-f4bf-2a77-6c01-deaffb12a77c} {Suffrage:Voter ID:server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f Address:0e7e1a2b-8d79-b5c3-987f-500ec4aed88f} {Suffrage:Voter ID:server-d3407936-5652-58ee-f84c-42d117f8e408 Address:d3407936-5652-58ee-f84c-42d117f8e408} {Suffrage:Voter ID:server-b36ee052-17b3-fd43-7ff2-3d5138893f00 Address:b36ee052-17b3-fd43-7ff2-3d5138893f00} {Suffrage:Voter ID:server-e6b7053d-26a2-9666-7c6c-317faf67c173 Address:e6b7053d-26a2-9666-7c6c-317faf67c173} {Suffrage:Voter ID:server-2a7a923b-38ae-bd9f-3b81-3062a5006378 Address:2a7a923b-38ae-bd9f-3b81-3062a5006378} {Suffrage:Voter ID:server-7cab0268-da6c-6645-0c22-43dfdbcc39af Address:7cab0268-da6c-6645-0c22-43dfdbcc39af}]"
2024-05-07T18:36:12.061Z [INFO]  server-d3407936-5652-58ee-f84c-42d117f8e408: entering follower state: follower="Node at d3407936-5652-58ee-f84c-42d117f8e408 [Follower]" leader-address= leader-id=
2024-05-07T18:36:12.061Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cbec760-f4bf-2a77-6c01-deaffb12a77c Address:2cbec760-f4bf-2a77-6c01-deaffb12a77c} {Suffrage:Voter ID:server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f Address:0e7e1a2b-8d79-b5c3-987f-500ec4aed88f} {Suffrage:Voter ID:server-d3407936-5652-58ee-f84c-42d117f8e408 Address:d3407936-5652-58ee-f84c-42d117f8e408} {Suffrage:Voter ID:server-b36ee052-17b3-fd43-7ff2-3d5138893f00 Address:b36ee052-17b3-fd43-7ff2-3d5138893f00} {Suffrage:Voter ID:server-e6b7053d-26a2-9666-7c6c-317faf67c173 Address:e6b7053d-26a2-9666-7c6c-317faf67c173} {Suffrage:Voter ID:server-2a7a923b-38ae-bd9f-3b81-3062a5006378 Address:2a7a923b-38ae-bd9f-3b81-3062a5006378} {Suffrage:Voter ID:server-7cab0268-da6c-6645-0c22-43dfdbcc39af Address:7cab0268-da6c-6645-0c22-43dfdbcc39af}]"
2024-05-07T18:36:12.061Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: entering follower state: follower="Node at b36ee052-17b3-fd43-7ff2-3d5138893f00 [Follower]" leader-address= leader-id=
2024-05-07T18:36:12.061Z [INFO]  server-e6b7053d-26a2-9666-7c6c-317faf67c173: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cbec760-f4bf-2a77-6c01-deaffb12a77c Address:2cbec760-f4bf-2a77-6c01-deaffb12a77c} {Suffrage:Voter ID:server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f Address:0e7e1a2b-8d79-b5c3-987f-500ec4aed88f} {Suffrage:Voter ID:server-d3407936-5652-58ee-f84c-42d117f8e408 Address:d3407936-5652-58ee-f84c-42d117f8e408} {Suffrage:Voter ID:server-b36ee052-17b3-fd43-7ff2-3d5138893f00 Address:b36ee052-17b3-fd43-7ff2-3d5138893f00} {Suffrage:Voter ID:server-e6b7053d-26a2-9666-7c6c-317faf67c173 Address:e6b7053d-26a2-9666-7c6c-317faf67c173} {Suffrage:Voter ID:server-2a7a923b-38ae-bd9f-3b81-3062a5006378 Address:2a7a923b-38ae-bd9f-3b81-3062a5006378} {Suffrage:Voter ID:server-7cab0268-da6c-6645-0c22-43dfdbcc39af Address:7cab0268-da6c-6645-0c22-43dfdbcc39af}]"
2024-05-07T18:36:12.061Z [INFO]  server-e6b7053d-26a2-9666-7c6c-317faf67c173: entering follower state: follower="Node at e6b7053d-26a2-9666-7c6c-317faf67c173 [Follower]" leader-address= leader-id=
2024-05-07T18:36:12.061Z [INFO]  server-2a7a923b-38ae-bd9f-3b81-3062a5006378: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cbec760-f4bf-2a77-6c01-deaffb12a77c Address:2cbec760-f4bf-2a77-6c01-deaffb12a77c} {Suffrage:Voter ID:server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f Address:0e7e1a2b-8d79-b5c3-987f-500ec4aed88f} {Suffrage:Voter ID:server-d3407936-5652-58ee-f84c-42d117f8e408 Address:d3407936-5652-58ee-f84c-42d117f8e408} {Suffrage:Voter ID:server-b36ee052-17b3-fd43-7ff2-3d5138893f00 Address:b36ee052-17b3-fd43-7ff2-3d5138893f00} {Suffrage:Voter ID:server-e6b7053d-26a2-9666-7c6c-317faf67c173 Address:e6b7053d-26a2-9666-7c6c-317faf67c173} {Suffrage:Voter ID:server-2a7a923b-38ae-bd9f-3b81-3062a5006378 Address:2a7a923b-38ae-bd9f-3b81-3062a5006378} {Suffrage:Voter ID:server-7cab0268-da6c-6645-0c22-43dfdbcc39af Address:7cab0268-da6c-6645-0c22-43dfdbcc39af}]"
2024-05-07T18:36:12.062Z [INFO]  server-2a7a923b-38ae-bd9f-3b81-3062a5006378: entering follower state: follower="Node at 2a7a923b-38ae-bd9f-3b81-3062a5006378 [Follower]" leader-address= leader-id=
2024-05-07T18:36:12.062Z [INFO]  server-7cab0268-da6c-6645-0c22-43dfdbcc39af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cbec760-f4bf-2a77-6c01-deaffb12a77c Address:2cbec760-f4bf-2a77-6c01-deaffb12a77c} {Suffrage:Voter ID:server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f Address:0e7e1a2b-8d79-b5c3-987f-500ec4aed88f} {Suffrage:Voter ID:server-d3407936-5652-58ee-f84c-42d117f8e408 Address:d3407936-5652-58ee-f84c-42d117f8e408} {Suffrage:Voter ID:server-b36ee052-17b3-fd43-7ff2-3d5138893f00 Address:b36ee052-17b3-fd43-7ff2-3d5138893f00} {Suffrage:Voter ID:server-e6b7053d-26a2-9666-7c6c-317faf67c173 Address:e6b7053d-26a2-9666-7c6c-317faf67c173} {Suffrage:Voter ID:server-2a7a923b-38ae-bd9f-3b81-3062a5006378 Address:2a7a923b-38ae-bd9f-3b81-3062a5006378} {Suffrage:Voter ID:server-7cab0268-da6c-6645-0c22-43dfdbcc39af Address:7cab0268-da6c-6645-0c22-43dfdbcc39af}]"
2024-05-07T18:36:12.062Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:12.062Z [INFO]  server-7cab0268-da6c-6645-0c22-43dfdbcc39af: entering follower state: follower="Node at 7cab0268-da6c-6645-0c22-43dfdbcc39af [Follower]" leader-address= leader-id=
2024-05-07T18:36:12.121Z [WARN]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: entering candidate state: node="Node at b36ee052-17b3-fd43-7ff2-3d5138893f00 [Candidate]" term=2
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: election won: term=2 tally=4
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: entering leader state: leader="Node at b36ee052-17b3-fd43-7ff2-3d5138893f00 [Leader]"
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: added peer, starting replication: peer=server-2cbec760-f4bf-2a77-6c01-deaffb12a77c
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: added peer, starting replication: peer=server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: added peer, starting replication: peer=server-d3407936-5652-58ee-f84c-42d117f8e408
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: added peer, starting replication: peer=server-e6b7053d-26a2-9666-7c6c-317faf67c173
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: added peer, starting replication: peer=server-2a7a923b-38ae-bd9f-3b81-3062a5006378
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: added peer, starting replication: peer=server-7cab0268-da6c-6645-0c22-43dfdbcc39af
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: pipelining replication: peer="{Voter server-2cbec760-f4bf-2a77-6c01-deaffb12a77c 2cbec760-f4bf-2a77-6c01-deaffb12a77c}"
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: pipelining replication: peer="{Voter server-d3407936-5652-58ee-f84c-42d117f8e408 d3407936-5652-58ee-f84c-42d117f8e408}"
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: pipelining replication: peer="{Voter server-7cab0268-da6c-6645-0c22-43dfdbcc39af 7cab0268-da6c-6645-0c22-43dfdbcc39af}"
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: pipelining replication: peer="{Voter server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f 0e7e1a2b-8d79-b5c3-987f-500ec4aed88f}"
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: pipelining replication: peer="{Voter server-e6b7053d-26a2-9666-7c6c-317faf67c173 e6b7053d-26a2-9666-7c6c-317faf67c173}"
2024-05-07T18:36:12.121Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: pipelining replication: peer="{Voter server-2a7a923b-38ae-bd9f-3b81-3062a5006378 2a7a923b-38ae-bd9f-3b81-3062a5006378}"
2024-05-07T18:36:12.228Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:12.121882198 +0000 UTC m=+109.723000448 (1 nodes), 59.799424ms from start of poll, 63.46402ms from cluster start. Timeout at 2024-05-07 18:36:12.228008657 +0000 UTC m=+109.829126907, 106.126459ms after stability
2024-05-07T18:36:12.228Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:12.333Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:12.228186076 +0000 UTC m=+109.829304326 (6 nodes), 1.583µs from start of poll, 169.767898ms from cluster start. Timeout at 2024-05-07 18:36:12.333510875 +0000 UTC m=+109.934629125, 105.324799ms after stability
2024-05-07T18:36:12.333Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:12.439Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:12.333596644 +0000 UTC m=+109.934714894 (1 nodes), 2.224µs from start of poll, 275.178466ms from cluster start. Timeout at 2024-05-07 18:36:12.439290329 +0000 UTC m=+110.040408579, 105.693685ms after stability
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: entering candidate state: node="Node at 2cbec760-f4bf-2a77-6c01-deaffb12a77c [Candidate]" term=3
2024-05-07T18:36:12.439Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: entering follower state: follower="Node at b36ee052-17b3-fd43-7ff2-3d5138893f00 [Follower]" leader-address= leader-id=
2024-05-07T18:36:12.439Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: aborting pipeline replication: peer="{Voter server-2cbec760-f4bf-2a77-6c01-deaffb12a77c 2cbec760-f4bf-2a77-6c01-deaffb12a77c}"
2024-05-07T18:36:12.439Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: election won: term=3 tally=4
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: entering leader state: leader="Node at 2cbec760-f4bf-2a77-6c01-deaffb12a77c [Leader]"
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: added peer, starting replication: peer=server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: added peer, starting replication: peer=server-d3407936-5652-58ee-f84c-42d117f8e408
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: added peer, starting replication: peer=server-b36ee052-17b3-fd43-7ff2-3d5138893f00
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: added peer, starting replication: peer=server-e6b7053d-26a2-9666-7c6c-317faf67c173
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: added peer, starting replication: peer=server-2a7a923b-38ae-bd9f-3b81-3062a5006378
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: added peer, starting replication: peer=server-7cab0268-da6c-6645-0c22-43dfdbcc39af
2024-05-07T18:36:12.439Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: aborting pipeline replication: peer="{Voter server-7cab0268-da6c-6645-0c22-43dfdbcc39af 7cab0268-da6c-6645-0c22-43dfdbcc39af}"
2024-05-07T18:36:12.439Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: aborting pipeline replication: peer="{Voter server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f 0e7e1a2b-8d79-b5c3-987f-500ec4aed88f}"
2024-05-07T18:36:12.439Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: aborting pipeline replication: peer="{Voter server-d3407936-5652-58ee-f84c-42d117f8e408 d3407936-5652-58ee-f84c-42d117f8e408}"
2024-05-07T18:36:12.439Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: aborting pipeline replication: peer="{Voter server-e6b7053d-26a2-9666-7c6c-317faf67c173 e6b7053d-26a2-9666-7c6c-317faf67c173}"
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: pipelining replication: peer="{Voter server-7cab0268-da6c-6645-0c22-43dfdbcc39af 7cab0268-da6c-6645-0c22-43dfdbcc39af}"
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: pipelining replication: peer="{Voter server-e6b7053d-26a2-9666-7c6c-317faf67c173 e6b7053d-26a2-9666-7c6c-317faf67c173}"
2024-05-07T18:36:12.439Z [INFO]  server-b36ee052-17b3-fd43-7ff2-3d5138893f00: aborting pipeline replication: peer="{Voter server-2a7a923b-38ae-bd9f-3b81-3062a5006378 2a7a923b-38ae-bd9f-3b81-3062a5006378}"
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: pipelining replication: peer="{Voter server-d3407936-5652-58ee-f84c-42d117f8e408 d3407936-5652-58ee-f84c-42d117f8e408}"
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: pipelining replication: peer="{Voter server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f 0e7e1a2b-8d79-b5c3-987f-500ec4aed88f}"
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: pipelining replication: peer="{Voter server-2a7a923b-38ae-bd9f-3b81-3062a5006378 2a7a923b-38ae-bd9f-3b81-3062a5006378}"
2024-05-07T18:36:12.439Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: pipelining replication: peer="{Voter server-b36ee052-17b3-fd43-7ff2-3d5138893f00 b36ee052-17b3-fd43-7ff2-3d5138893f00}"
2024-05-07T18:36:12.545Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:12.439656921 +0000 UTC m=+110.040775181 (1 nodes), 174.375µs from start of poll, 381.238753ms from cluster start. Timeout at 2024-05-07 18:36:12.545474695 +0000 UTC m=+110.146592945, 105.817764ms after stability
2024-05-07T18:36:12.545Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: aborting pipeline replication: peer="{Voter server-7cab0268-da6c-6645-0c22-43dfdbcc39af 7cab0268-da6c-6645-0c22-43dfdbcc39af}"
2024-05-07T18:36:12.545Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: aborting pipeline replication: peer="{Voter server-0e7e1a2b-8d79-b5c3-987f-500ec4aed88f 0e7e1a2b-8d79-b5c3-987f-500ec4aed88f}"
2024-05-07T18:36:12.545Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: aborting pipeline replication: peer="{Voter server-d3407936-5652-58ee-f84c-42d117f8e408 d3407936-5652-58ee-f84c-42d117f8e408}"
2024-05-07T18:36:12.545Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: aborting pipeline replication: peer="{Voter server-b36ee052-17b3-fd43-7ff2-3d5138893f00 b36ee052-17b3-fd43-7ff2-3d5138893f00}"
2024-05-07T18:36:12.545Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: aborting pipeline replication: peer="{Voter server-e6b7053d-26a2-9666-7c6c-317faf67c173 e6b7053d-26a2-9666-7c6c-317faf67c173}"
2024-05-07T18:36:12.545Z [INFO]  server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: aborting pipeline replication: peer="{Voter server-2a7a923b-38ae-bd9f-3b81-3062a5006378 2a7a923b-38ae-bd9f-3b81-3062a5006378}"
2024-05-07T18:36:13.046Z [ERROR] server-2cbec760-f4bf-2a77-6c01-deaffb12a77c: failed to heartbeat to: peer=b36ee052-17b3-fd43-7ff2-3d5138893f00 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.99s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2024-05-07T18:36:13.047Z [INFO]  server-e879d615-ecfb-a3e9-13bf-6173482b18f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e879d615-ecfb-a3e9-13bf-6173482b18f3 Address:e879d615-ecfb-a3e9-13bf-6173482b18f3} {Suffrage:Voter ID:server-4146c413-a99e-2a5b-c167-b48020cfd7b2 Address:4146c413-a99e-2a5b-c167-b48020cfd7b2} {Suffrage:Voter ID:server-de9eadbc-7aca-3a21-32c3-66524c35e714 Address:de9eadbc-7aca-3a21-32c3-66524c35e714}]"
2024-05-07T18:36:13.047Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e879d615-ecfb-a3e9-13bf-6173482b18f3 Address:e879d615-ecfb-a3e9-13bf-6173482b18f3} {Suffrage:Voter ID:server-4146c413-a99e-2a5b-c167-b48020cfd7b2 Address:4146c413-a99e-2a5b-c167-b48020cfd7b2} {Suffrage:Voter ID:server-de9eadbc-7aca-3a21-32c3-66524c35e714 Address:de9eadbc-7aca-3a21-32c3-66524c35e714}]"
2024-05-07T18:36:13.047Z [INFO]  server-de9eadbc-7aca-3a21-32c3-66524c35e714: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e879d615-ecfb-a3e9-13bf-6173482b18f3 Address:e879d615-ecfb-a3e9-13bf-6173482b18f3} {Suffrage:Voter ID:server-4146c413-a99e-2a5b-c167-b48020cfd7b2 Address:4146c413-a99e-2a5b-c167-b48020cfd7b2} {Suffrage:Voter ID:server-de9eadbc-7aca-3a21-32c3-66524c35e714 Address:de9eadbc-7aca-3a21-32c3-66524c35e714}]"
2024-05-07T18:36:13.047Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:13.047Z [INFO]  server-e879d615-ecfb-a3e9-13bf-6173482b18f3: entering follower state: follower="Node at e879d615-ecfb-a3e9-13bf-6173482b18f3 [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.047Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: entering follower state: follower="Node at 4146c413-a99e-2a5b-c167-b48020cfd7b2 [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.047Z [INFO]  server-de9eadbc-7aca-3a21-32c3-66524c35e714: entering follower state: follower="Node at de9eadbc-7aca-3a21-32c3-66524c35e714 [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.116Z [WARN]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:13.116Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: entering candidate state: node="Node at 4146c413-a99e-2a5b-c167-b48020cfd7b2 [Candidate]" term=2
2024-05-07T18:36:13.117Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: election won: term=2 tally=2
2024-05-07T18:36:13.117Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: entering leader state: leader="Node at 4146c413-a99e-2a5b-c167-b48020cfd7b2 [Leader]"
2024-05-07T18:36:13.117Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: added peer, starting replication: peer=server-e879d615-ecfb-a3e9-13bf-6173482b18f3
2024-05-07T18:36:13.117Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: added peer, starting replication: peer=server-de9eadbc-7aca-3a21-32c3-66524c35e714
2024-05-07T18:36:13.117Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: pipelining replication: peer="{Voter server-de9eadbc-7aca-3a21-32c3-66524c35e714 de9eadbc-7aca-3a21-32c3-66524c35e714}"
2024-05-07T18:36:13.117Z [WARN]  server-de9eadbc-7aca-3a21-32c3-66524c35e714: rejecting vote request since our last term is greater: candidate=4146c413-a99e-2a5b-c167-b48020cfd7b2 last-term=2 last-candidate-term=1
2024-05-07T18:36:13.117Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: pipelining replication: peer="{Voter server-e879d615-ecfb-a3e9-13bf-6173482b18f3 e879d615-ecfb-a3e9-13bf-6173482b18f3}"
2024-05-07T18:36:13.222Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:13.117241149 +0000 UTC m=+110.718359409 (1 nodes), 69.445895ms from start of poll, 69.75516ms from cluster start. Timeout at 2024-05-07 18:36:13.222837182 +0000 UTC m=+110.823955432, 105.596023ms after stability
2024-05-07T18:36:13.222Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: aborting pipeline replication: peer="{Voter server-de9eadbc-7aca-3a21-32c3-66524c35e714 de9eadbc-7aca-3a21-32c3-66524c35e714}"
2024-05-07T18:36:13.223Z [INFO]  server-4146c413-a99e-2a5b-c167-b48020cfd7b2: aborting pipeline replication: peer="{Voter server-e879d615-ecfb-a3e9-13bf-6173482b18f3 e879d615-ecfb-a3e9-13bf-6173482b18f3}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.18s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2024-05-07T18:36:13.223Z [INFO]  server-2c0b170c-2ff6-6c3d-4433-62b7046a0adf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c0b170c-2ff6-6c3d-4433-62b7046a0adf Address:2c0b170c-2ff6-6c3d-4433-62b7046a0adf} {Suffrage:Voter ID:server-49b0b0b4-af58-e5cd-beb3-fa7afa022697 Address:49b0b0b4-af58-e5cd-beb3-fa7afa022697} {Suffrage:Voter ID:server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c Address:4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c}]"
2024-05-07T18:36:13.224Z [INFO]  server-49b0b0b4-af58-e5cd-beb3-fa7afa022697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c0b170c-2ff6-6c3d-4433-62b7046a0adf Address:2c0b170c-2ff6-6c3d-4433-62b7046a0adf} {Suffrage:Voter ID:server-49b0b0b4-af58-e5cd-beb3-fa7afa022697 Address:49b0b0b4-af58-e5cd-beb3-fa7afa022697} {Suffrage:Voter ID:server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c Address:4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c}]"
2024-05-07T18:36:13.224Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c0b170c-2ff6-6c3d-4433-62b7046a0adf Address:2c0b170c-2ff6-6c3d-4433-62b7046a0adf} {Suffrage:Voter ID:server-49b0b0b4-af58-e5cd-beb3-fa7afa022697 Address:49b0b0b4-af58-e5cd-beb3-fa7afa022697} {Suffrage:Voter ID:server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c Address:4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c}]"
2024-05-07T18:36:13.224Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:13.224Z [INFO]  server-2c0b170c-2ff6-6c3d-4433-62b7046a0adf: entering follower state: follower="Node at 2c0b170c-2ff6-6c3d-4433-62b7046a0adf [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.224Z [INFO]  server-49b0b0b4-af58-e5cd-beb3-fa7afa022697: entering follower state: follower="Node at 49b0b0b4-af58-e5cd-beb3-fa7afa022697 [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.224Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: entering follower state: follower="Node at 4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.302Z [WARN]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:13.302Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: entering candidate state: node="Node at 4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c [Candidate]" term=2
2024-05-07T18:36:13.302Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: election won: term=2 tally=2
2024-05-07T18:36:13.302Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: entering leader state: leader="Node at 4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c [Leader]"
2024-05-07T18:36:13.302Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: added peer, starting replication: peer=server-2c0b170c-2ff6-6c3d-4433-62b7046a0adf
2024-05-07T18:36:13.302Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: added peer, starting replication: peer=server-49b0b0b4-af58-e5cd-beb3-fa7afa022697
2024-05-07T18:36:13.302Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: pipelining replication: peer="{Voter server-49b0b0b4-af58-e5cd-beb3-fa7afa022697 49b0b0b4-af58-e5cd-beb3-fa7afa022697}"
2024-05-07T18:36:13.302Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: pipelining replication: peer="{Voter server-2c0b170c-2ff6-6c3d-4433-62b7046a0adf 2c0b170c-2ff6-6c3d-4433-62b7046a0adf}"
2024-05-07T18:36:13.407Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:13.302495161 +0000 UTC m=+110.903613411 (2 nodes), 78.42212ms from start of poll, 78.662628ms from cluster start. Timeout at 2024-05-07 18:36:13.407498282 +0000 UTC m=+111.008616532, 105.003121ms after stability
2024-05-07T18:36:13.407Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:13.512Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:13.407548976 +0000 UTC m=+111.008667226 (1 nodes), 922ns from start of poll, 183.716443ms from cluster start. Timeout at 2024-05-07 18:36:13.512566482 +0000 UTC m=+111.113684722, 105.017496ms after stability
2024-05-07T18:36:13.512Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: aborting pipeline replication: peer="{Voter server-49b0b0b4-af58-e5cd-beb3-fa7afa022697 49b0b0b4-af58-e5cd-beb3-fa7afa022697}"
2024-05-07T18:36:13.512Z [INFO]  server-4ab9c2e1-9a2c-37ac-8c6a-c8e32f0b829c: aborting pipeline replication: peer="{Voter server-2c0b170c-2ff6-6c3d-4433-62b7046a0adf 2c0b170c-2ff6-6c3d-4433-62b7046a0adf}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2024-05-07T18:36:13.513Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71047e26-0060-e7e4-6ded-9390fc72ec0b Address:71047e26-0060-e7e4-6ded-9390fc72ec0b} {Suffrage:Voter ID:server-afc57c73-6873-7619-b891-2b86a0cd6dab Address:afc57c73-6873-7619-b891-2b86a0cd6dab} {Suffrage:Voter ID:server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d Address:e3944f7b-537c-3e29-0856-3cc1ce9e7d5d}]"
2024-05-07T18:36:13.513Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71047e26-0060-e7e4-6ded-9390fc72ec0b Address:71047e26-0060-e7e4-6ded-9390fc72ec0b} {Suffrage:Voter ID:server-afc57c73-6873-7619-b891-2b86a0cd6dab Address:afc57c73-6873-7619-b891-2b86a0cd6dab} {Suffrage:Voter ID:server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d Address:e3944f7b-537c-3e29-0856-3cc1ce9e7d5d}]"
2024-05-07T18:36:13.513Z [INFO]  server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71047e26-0060-e7e4-6ded-9390fc72ec0b Address:71047e26-0060-e7e4-6ded-9390fc72ec0b} {Suffrage:Voter ID:server-afc57c73-6873-7619-b891-2b86a0cd6dab Address:afc57c73-6873-7619-b891-2b86a0cd6dab} {Suffrage:Voter ID:server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d Address:e3944f7b-537c-3e29-0856-3cc1ce9e7d5d}]"
2024-05-07T18:36:13.513Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:13.513Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: entering follower state: follower="Node at 71047e26-0060-e7e4-6ded-9390fc72ec0b [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.513Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: entering follower state: follower="Node at afc57c73-6873-7619-b891-2b86a0cd6dab [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.513Z [INFO]  server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d: entering follower state: follower="Node at e3944f7b-537c-3e29-0856-3cc1ce9e7d5d [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.588Z [WARN]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:13.588Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: entering candidate state: node="Node at afc57c73-6873-7619-b891-2b86a0cd6dab [Candidate]" term=2
2024-05-07T18:36:13.588Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: election won: term=2 tally=2
2024-05-07T18:36:13.588Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: entering leader state: leader="Node at afc57c73-6873-7619-b891-2b86a0cd6dab [Leader]"
2024-05-07T18:36:13.588Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: added peer, starting replication: peer=server-71047e26-0060-e7e4-6ded-9390fc72ec0b
2024-05-07T18:36:13.588Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: added peer, starting replication: peer=server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d
2024-05-07T18:36:13.588Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: pipelining replication: peer="{Voter server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d e3944f7b-537c-3e29-0856-3cc1ce9e7d5d}"
2024-05-07T18:36:13.589Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: pipelining replication: peer="{Voter server-71047e26-0060-e7e4-6ded-9390fc72ec0b 71047e26-0060-e7e4-6ded-9390fc72ec0b}"
2024-05-07T18:36:13.694Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:13.58911621 +0000 UTC m=+111.190234460 (1 nodes), 75.345067ms from start of poll, 75.603557ms from cluster start. Timeout at 2024-05-07 18:36:13.694183059 +0000 UTC m=+111.295301309, 105.066849ms after stability
2024-05-07T18:36:13.694Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:13.799Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:13.694255724 +0000 UTC m=+111.295373974 (2 nodes), 2.595µs from start of poll, 180.743071ms from cluster start. Timeout at 2024-05-07 18:36:13.799933479 +0000 UTC m=+111.401051729, 105.677755ms after stability
2024-05-07T18:36:13.830Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: entering candidate state: node="Node at 71047e26-0060-e7e4-6ded-9390fc72ec0b [Candidate]" term=3
2024-05-07T18:36:13.830Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:13.830Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: entering follower state: follower="Node at afc57c73-6873-7619-b891-2b86a0cd6dab [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.830Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: aborting pipeline replication: peer="{Voter server-71047e26-0060-e7e4-6ded-9390fc72ec0b 71047e26-0060-e7e4-6ded-9390fc72ec0b}"
2024-05-07T18:36:13.830Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: election won: term=3 tally=2
2024-05-07T18:36:13.830Z [INFO]  server-afc57c73-6873-7619-b891-2b86a0cd6dab: aborting pipeline replication: peer="{Voter server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d e3944f7b-537c-3e29-0856-3cc1ce9e7d5d}"
2024-05-07T18:36:13.830Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: entering leader state: leader="Node at 71047e26-0060-e7e4-6ded-9390fc72ec0b [Leader]"
2024-05-07T18:36:13.830Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: added peer, starting replication: peer=server-afc57c73-6873-7619-b891-2b86a0cd6dab
2024-05-07T18:36:13.830Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: added peer, starting replication: peer=server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d
2024-05-07T18:36:13.830Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: pipelining replication: peer="{Voter server-afc57c73-6873-7619-b891-2b86a0cd6dab afc57c73-6873-7619-b891-2b86a0cd6dab}"
2024-05-07T18:36:13.830Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: pipelining replication: peer="{Voter server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d e3944f7b-537c-3e29-0856-3cc1ce9e7d5d}"
2024-05-07T18:36:13.935Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:13.830663325 +0000 UTC m=+111.431781575 (1 nodes), 97.943µs from start of poll, 317.150672ms from cluster start. Timeout at 2024-05-07 18:36:13.935934704 +0000 UTC m=+111.537052954, 105.271379ms after stability
2024-05-07T18:36:13.936Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: aborting pipeline replication: peer="{Voter server-afc57c73-6873-7619-b891-2b86a0cd6dab afc57c73-6873-7619-b891-2b86a0cd6dab}"
2024-05-07T18:36:13.936Z [INFO]  server-71047e26-0060-e7e4-6ded-9390fc72ec0b: aborting pipeline replication: peer="{Voter server-e3944f7b-537c-3e29-0856-3cc1ce9e7d5d e3944f7b-537c-3e29-0856-3cc1ce9e7d5d}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s)
=== RUN   TestRaft_LeadershipTransferToItself
2024-05-07T18:36:13.939Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de294b3c-6a8c-d684-e88d-b7fc159eef4e Address:de294b3c-6a8c-d684-e88d-b7fc159eef4e} {Suffrage:Voter ID:server-a20b5208-c334-3899-92a7-ac390375ac57 Address:a20b5208-c334-3899-92a7-ac390375ac57} {Suffrage:Voter ID:server-62e468fc-9e25-2a22-f50e-e61beb40eafe Address:62e468fc-9e25-2a22-f50e-e61beb40eafe}]"
2024-05-07T18:36:13.939Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: entering follower state: follower="Node at de294b3c-6a8c-d684-e88d-b7fc159eef4e [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.939Z [INFO]  server-a20b5208-c334-3899-92a7-ac390375ac57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de294b3c-6a8c-d684-e88d-b7fc159eef4e Address:de294b3c-6a8c-d684-e88d-b7fc159eef4e} {Suffrage:Voter ID:server-a20b5208-c334-3899-92a7-ac390375ac57 Address:a20b5208-c334-3899-92a7-ac390375ac57} {Suffrage:Voter ID:server-62e468fc-9e25-2a22-f50e-e61beb40eafe Address:62e468fc-9e25-2a22-f50e-e61beb40eafe}]"
2024-05-07T18:36:13.939Z [INFO]  server-a20b5208-c334-3899-92a7-ac390375ac57: entering follower state: follower="Node at a20b5208-c334-3899-92a7-ac390375ac57 [Follower]" leader-address= leader-id=
2024-05-07T18:36:13.939Z [INFO]  server-62e468fc-9e25-2a22-f50e-e61beb40eafe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de294b3c-6a8c-d684-e88d-b7fc159eef4e Address:de294b3c-6a8c-d684-e88d-b7fc159eef4e} {Suffrage:Voter ID:server-a20b5208-c334-3899-92a7-ac390375ac57 Address:a20b5208-c334-3899-92a7-ac390375ac57} {Suffrage:Voter ID:server-62e468fc-9e25-2a22-f50e-e61beb40eafe Address:62e468fc-9e25-2a22-f50e-e61beb40eafe}]"
2024-05-07T18:36:13.939Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:13.939Z [INFO]  server-62e468fc-9e25-2a22-f50e-e61beb40eafe: entering follower state: follower="Node at 62e468fc-9e25-2a22-f50e-e61beb40eafe [Follower]" leader-address= leader-id=
2024-05-07T18:36:14.008Z [WARN]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:14.008Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: entering candidate state: node="Node at de294b3c-6a8c-d684-e88d-b7fc159eef4e [Candidate]" term=2
2024-05-07T18:36:14.008Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: election won: term=2 tally=2
2024-05-07T18:36:14.008Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: entering leader state: leader="Node at de294b3c-6a8c-d684-e88d-b7fc159eef4e [Leader]"
2024-05-07T18:36:14.008Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: added peer, starting replication: peer=server-a20b5208-c334-3899-92a7-ac390375ac57
2024-05-07T18:36:14.008Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: added peer, starting replication: peer=server-62e468fc-9e25-2a22-f50e-e61beb40eafe
2024-05-07T18:36:14.008Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: pipelining replication: peer="{Voter server-62e468fc-9e25-2a22-f50e-e61beb40eafe 62e468fc-9e25-2a22-f50e-e61beb40eafe}"
2024-05-07T18:36:14.008Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: pipelining replication: peer="{Voter server-a20b5208-c334-3899-92a7-ac390375ac57 a20b5208-c334-3899-92a7-ac390375ac57}"
2024-05-07T18:36:14.118Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:14.008526621 +0000 UTC m=+111.609644881 (1 nodes), 68.971013ms from start of poll, 69.299414ms from cluster start. Timeout at 2024-05-07 18:36:14.118075605 +0000 UTC m=+111.719193855, 109.548974ms after stability
2024-05-07T18:36:14.118Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: cannot transfer leadership to itself
2024-05-07T18:36:14.118Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: aborting pipeline replication: peer="{Voter server-62e468fc-9e25-2a22-f50e-e61beb40eafe 62e468fc-9e25-2a22-f50e-e61beb40eafe}"
2024-05-07T18:36:14.118Z [INFO]  server-de294b3c-6a8c-d684-e88d-b7fc159eef4e: aborting pipeline replication: peer="{Voter server-a20b5208-c334-3899-92a7-ac390375ac57 a20b5208-c334-3899-92a7-ac390375ac57}"
--- PASS: TestRaft_LeadershipTransferToItself (0.18s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2024-05-07T18:36:14.119Z [INFO]  server-59f67d04-30f5-c63e-7d57-46279ebf7326: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59f67d04-30f5-c63e-7d57-46279ebf7326 Address:59f67d04-30f5-c63e-7d57-46279ebf7326} {Suffrage:Voter ID:server-28d94de0-9782-b318-1b80-406f951fff4a Address:28d94de0-9782-b318-1b80-406f951fff4a} {Suffrage:Voter ID:server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79 Address:dca49ed4-5b79-b3c8-ed8d-3bb675babf79}]"
2024-05-07T18:36:14.119Z [INFO]  server-59f67d04-30f5-c63e-7d57-46279ebf7326: entering follower state: follower="Node at 59f67d04-30f5-c63e-7d57-46279ebf7326 [Follower]" leader-address= leader-id=
2024-05-07T18:36:14.119Z [INFO]  server-28d94de0-9782-b318-1b80-406f951fff4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59f67d04-30f5-c63e-7d57-46279ebf7326 Address:59f67d04-30f5-c63e-7d57-46279ebf7326} {Suffrage:Voter ID:server-28d94de0-9782-b318-1b80-406f951fff4a Address:28d94de0-9782-b318-1b80-406f951fff4a} {Suffrage:Voter ID:server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79 Address:dca49ed4-5b79-b3c8-ed8d-3bb675babf79}]"
2024-05-07T18:36:14.119Z [INFO]  server-28d94de0-9782-b318-1b80-406f951fff4a: entering follower state: follower="Node at 28d94de0-9782-b318-1b80-406f951fff4a [Follower]" leader-address= leader-id=
2024-05-07T18:36:14.119Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59f67d04-30f5-c63e-7d57-46279ebf7326 Address:59f67d04-30f5-c63e-7d57-46279ebf7326} {Suffrage:Voter ID:server-28d94de0-9782-b318-1b80-406f951fff4a Address:28d94de0-9782-b318-1b80-406f951fff4a} {Suffrage:Voter ID:server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79 Address:dca49ed4-5b79-b3c8-ed8d-3bb675babf79}]"
2024-05-07T18:36:14.119Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:14.119Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: entering follower state: follower="Node at dca49ed4-5b79-b3c8-ed8d-3bb675babf79 [Follower]" leader-address= leader-id=
2024-05-07T18:36:14.177Z [WARN]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:14.177Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: entering candidate state: node="Node at dca49ed4-5b79-b3c8-ed8d-3bb675babf79 [Candidate]" term=2
2024-05-07T18:36:14.177Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: election won: term=2 tally=2
2024-05-07T18:36:14.177Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: entering leader state: leader="Node at dca49ed4-5b79-b3c8-ed8d-3bb675babf79 [Leader]"
2024-05-07T18:36:14.177Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: added peer, starting replication: peer=server-59f67d04-30f5-c63e-7d57-46279ebf7326
2024-05-07T18:36:14.177Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: added peer, starting replication: peer=server-28d94de0-9782-b318-1b80-406f951fff4a
2024-05-07T18:36:14.178Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: pipelining replication: peer="{Voter server-28d94de0-9782-b318-1b80-406f951fff4a 28d94de0-9782-b318-1b80-406f951fff4a}"
2024-05-07T18:36:14.178Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: pipelining replication: peer="{Voter server-59f67d04-30f5-c63e-7d57-46279ebf7326 59f67d04-30f5-c63e-7d57-46279ebf7326}"
2024-05-07T18:36:14.283Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:14.178284831 +0000 UTC m=+111.779403081 (1 nodes), 58.351801ms from start of poll, 58.756713ms from cluster start. Timeout at 2024-05-07 18:36:14.283883779 +0000 UTC m=+111.885002029, 105.598948ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2024-05-07T18:36:14.284Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: aborting pipeline replication: peer="{Voter server-59f67d04-30f5-c63e-7d57-46279ebf7326 59f67d04-30f5-c63e-7d57-46279ebf7326}"
2024-05-07T18:36:14.284Z [INFO]  server-dca49ed4-5b79-b3c8-ed8d-3bb675babf79: aborting pipeline replication: peer="{Voter server-28d94de0-9782-b318-1b80-406f951fff4a 28d94de0-9782-b318-1b80-406f951fff4a}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2024-05-07T18:36:14.285Z [INFO]  server-beea7418-6955-0ef4-f527-01af8b551181: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-beea7418-6955-0ef4-f527-01af8b551181 Address:beea7418-6955-0ef4-f527-01af8b551181} {Suffrage:Voter ID:server-c902b00e-fadc-f378-6b77-50f238240ded Address:c902b00e-fadc-f378-6b77-50f238240ded} {Suffrage:Voter ID:server-eec50481-9c6d-302d-1adb-0cf1467dc584 Address:eec50481-9c6d-302d-1adb-0cf1467dc584}]"
2024-05-07T18:36:14.285Z [INFO]  server-c902b00e-fadc-f378-6b77-50f238240ded: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-beea7418-6955-0ef4-f527-01af8b551181 Address:beea7418-6955-0ef4-f527-01af8b551181} {Suffrage:Voter ID:server-c902b00e-fadc-f378-6b77-50f238240ded Address:c902b00e-fadc-f378-6b77-50f238240ded} {Suffrage:Voter ID:server-eec50481-9c6d-302d-1adb-0cf1467dc584 Address:eec50481-9c6d-302d-1adb-0cf1467dc584}]"
2024-05-07T18:36:14.285Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-beea7418-6955-0ef4-f527-01af8b551181 Address:beea7418-6955-0ef4-f527-01af8b551181} {Suffrage:Voter ID:server-c902b00e-fadc-f378-6b77-50f238240ded Address:c902b00e-fadc-f378-6b77-50f238240ded} {Suffrage:Voter ID:server-eec50481-9c6d-302d-1adb-0cf1467dc584 Address:eec50481-9c6d-302d-1adb-0cf1467dc584}]"
2024-05-07T18:36:14.285Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:14.285Z [INFO]  server-beea7418-6955-0ef4-f527-01af8b551181: entering follower state: follower="Node at beea7418-6955-0ef4-f527-01af8b551181 [Follower]" leader-address= leader-id=
2024-05-07T18:36:14.285Z [INFO]  server-c902b00e-fadc-f378-6b77-50f238240ded: entering follower state: follower="Node at c902b00e-fadc-f378-6b77-50f238240ded [Follower]" leader-address= leader-id=
2024-05-07T18:36:14.285Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: entering follower state: follower="Node at eec50481-9c6d-302d-1adb-0cf1467dc584 [Follower]" leader-address= leader-id=
2024-05-07T18:36:14.348Z [WARN]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:14.348Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: entering candidate state: node="Node at eec50481-9c6d-302d-1adb-0cf1467dc584 [Candidate]" term=2
2024-05-07T18:36:14.348Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: election won: term=2 tally=2
2024-05-07T18:36:14.348Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: entering leader state: leader="Node at eec50481-9c6d-302d-1adb-0cf1467dc584 [Leader]"
2024-05-07T18:36:14.348Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: added peer, starting replication: peer=server-beea7418-6955-0ef4-f527-01af8b551181
2024-05-07T18:36:14.348Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: added peer, starting replication: peer=server-c902b00e-fadc-f378-6b77-50f238240ded
2024-05-07T18:36:14.348Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: pipelining replication: peer="{Voter server-c902b00e-fadc-f378-6b77-50f238240ded c902b00e-fadc-f378-6b77-50f238240ded}"
2024-05-07T18:36:14.349Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: pipelining replication: peer="{Voter server-beea7418-6955-0ef4-f527-01af8b551181 beea7418-6955-0ef4-f527-01af8b551181}"
2024-05-07T18:36:14.454Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:14.349137287 +0000 UTC m=+111.950255537 (1 nodes), 64.003182ms from start of poll, 64.219925ms from cluster start. Timeout at 2024-05-07 18:36:14.454247786 +0000 UTC m=+112.055366036, 105.110499ms after stability
2024-05-07T18:36:14.454Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:14.559Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:14.45430313 +0000 UTC m=+112.055421380 (2 nodes), 1.753µs from start of poll, 169.385768ms from cluster start. Timeout at 2024-05-07 18:36:14.559519376 +0000 UTC m=+112.160637626, 105.216246ms after stability
2024-05-07T18:36:14.875Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: aborting pipeline replication: peer="{Voter server-c902b00e-fadc-f378-6b77-50f238240ded c902b00e-fadc-f378-6b77-50f238240ded}"
2024-05-07T18:36:14.875Z [INFO]  server-eec50481-9c6d-302d-1adb-0cf1467dc584: aborting pipeline replication: peer="{Voter server-beea7418-6955-0ef4-f527-01af8b551181 beea7418-6955-0ef4-f527-01af8b551181}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.59s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2024-05-07T18:36:14.875Z [INFO]  server-72cbda1a-b656-5210-13cb-6c999f3f8eb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72cbda1a-b656-5210-13cb-6c999f3f8eb8 Address:72cbda1a-b656-5210-13cb-6c999f3f8eb8} {Suffrage:Voter ID:server-e853e6b7-d292-7761-1fa5-e986e5d5c80b Address:e853e6b7-d292-7761-1fa5-e986e5d5c80b}]"
2024-05-07T18:36:14.875Z [INFO]  server-e853e6b7-d292-7761-1fa5-e986e5d5c80b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72cbda1a-b656-5210-13cb-6c999f3f8eb8 Address:72cbda1a-b656-5210-13cb-6c999f3f8eb8} {Suffrage:Voter ID:server-e853e6b7-d292-7761-1fa5-e986e5d5c80b Address:e853e6b7-d292-7761-1fa5-e986e5d5c80b}]"
2024-05-07T18:36:14.876Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:14.876Z [INFO]  server-72cbda1a-b656-5210-13cb-6c999f3f8eb8: entering follower state: follower="Node at 72cbda1a-b656-5210-13cb-6c999f3f8eb8 [Follower]" leader-address= leader-id=
2024-05-07T18:36:14.876Z [INFO]  server-e853e6b7-d292-7761-1fa5-e986e5d5c80b: entering follower state: follower="Node at e853e6b7-d292-7761-1fa5-e986e5d5c80b [Follower]" leader-address= leader-id=
2024-05-07T18:36:14.949Z [WARN]  server-72cbda1a-b656-5210-13cb-6c999f3f8eb8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:14.949Z [INFO]  server-72cbda1a-b656-5210-13cb-6c999f3f8eb8: entering candidate state: node="Node at 72cbda1a-b656-5210-13cb-6c999f3f8eb8 [Candidate]" term=2
2024-05-07T18:36:14.949Z [INFO]  server-72cbda1a-b656-5210-13cb-6c999f3f8eb8: election won: term=2 tally=2
2024-05-07T18:36:14.949Z [INFO]  server-72cbda1a-b656-5210-13cb-6c999f3f8eb8: entering leader state: leader="Node at 72cbda1a-b656-5210-13cb-6c999f3f8eb8 [Leader]"
2024-05-07T18:36:14.949Z [INFO]  server-72cbda1a-b656-5210-13cb-6c999f3f8eb8: added peer, starting replication: peer=server-e853e6b7-d292-7761-1fa5-e986e5d5c80b
2024-05-07T18:36:14.949Z [INFO]  server-72cbda1a-b656-5210-13cb-6c999f3f8eb8: pipelining replication: peer="{Voter server-e853e6b7-d292-7761-1fa5-e986e5d5c80b e853e6b7-d292-7761-1fa5-e986e5d5c80b}"
2024-05-07T18:36:15.055Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:14.949760883 +0000 UTC m=+112.550879133 (1 nodes), 73.746784ms from start of poll, 73.936627ms from cluster start. Timeout at 2024-05-07 18:36:15.055507847 +0000 UTC m=+112.656626097, 105.746964ms after stability
2024-05-07T18:36:15.055Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:15.161Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:15.055589409 +0000 UTC m=+112.656707659 (1 nodes), 1.052µs from start of poll, 179.765153ms from cluster start. Timeout at 2024-05-07 18:36:15.161127493 +0000 UTC m=+112.762245743, 105.538084ms after stability
2024-05-07T18:36:15.161Z [INFO]  server-72cbda1a-b656-5210-13cb-6c999f3f8eb8: updating configuration: command=DemoteVoter server-id=server-e853e6b7-d292-7761-1fa5-e986e5d5c80b server-addr= servers="[{Suffrage:Voter ID:server-72cbda1a-b656-5210-13cb-6c999f3f8eb8 Address:72cbda1a-b656-5210-13cb-6c999f3f8eb8} {Suffrage:Nonvoter ID:server-e853e6b7-d292-7761-1fa5-e986e5d5c80b Address:e853e6b7-d292-7761-1fa5-e986e5d5c80b}]"
2024-05-07T18:36:15.161Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:15.267Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:15.161302098 +0000 UTC m=+112.762420348 (1 nodes), 842ns from start of poll, 285.477842ms from cluster start. Timeout at 2024-05-07 18:36:15.267545956 +0000 UTC m=+112.868664196, 106.243848ms after stability
2024-05-07T18:36:15.267Z [INFO]  server-72cbda1a-b656-5210-13cb-6c999f3f8eb8: aborting pipeline replication: peer="{Voter server-e853e6b7-d292-7761-1fa5-e986e5d5c80b e853e6b7-d292-7761-1fa5-e986e5d5c80b}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.39s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2024-05-07T18:36:15.268Z [INFO]  server-78eea098-0044-92f9-df3f-fb4533c71060: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78eea098-0044-92f9-df3f-fb4533c71060 Address:78eea098-0044-92f9-df3f-fb4533c71060} {Suffrage:Voter ID:server-c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e Address:c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e}]"
2024-05-07T18:36:15.268Z [INFO]  server-c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78eea098-0044-92f9-df3f-fb4533c71060 Address:78eea098-0044-92f9-df3f-fb4533c71060} {Suffrage:Voter ID:server-c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e Address:c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e}]"
2024-05-07T18:36:15.268Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:15.268Z [INFO]  server-78eea098-0044-92f9-df3f-fb4533c71060: entering follower state: follower="Node at 78eea098-0044-92f9-df3f-fb4533c71060 [Follower]" leader-address= leader-id=
2024-05-07T18:36:15.268Z [INFO]  server-c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e: entering follower state: follower="Node at c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e [Follower]" leader-address= leader-id=
2024-05-07T18:36:15.319Z [WARN]  server-78eea098-0044-92f9-df3f-fb4533c71060: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:15.319Z [INFO]  server-78eea098-0044-92f9-df3f-fb4533c71060: entering candidate state: node="Node at 78eea098-0044-92f9-df3f-fb4533c71060 [Candidate]" term=2
2024-05-07T18:36:15.320Z [INFO]  server-78eea098-0044-92f9-df3f-fb4533c71060: election won: term=2 tally=2
2024-05-07T18:36:15.320Z [INFO]  server-78eea098-0044-92f9-df3f-fb4533c71060: entering leader state: leader="Node at 78eea098-0044-92f9-df3f-fb4533c71060 [Leader]"
2024-05-07T18:36:15.320Z [INFO]  server-78eea098-0044-92f9-df3f-fb4533c71060: added peer, starting replication: peer=server-c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e
2024-05-07T18:36:15.320Z [INFO]  server-78eea098-0044-92f9-df3f-fb4533c71060: pipelining replication: peer="{Voter server-c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e}"
2024-05-07T18:36:15.425Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:15.320141 +0000 UTC m=+112.921259250 (1 nodes), 51.518992ms from start of poll, 51.690412ms from cluster start. Timeout at 2024-05-07 18:36:15.425160259 +0000 UTC m=+113.026278509, 105.019259ms after stability
2024-05-07T18:36:15.425Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:15.530Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:15.425225891 +0000 UTC m=+113.026344131 (1 nodes), 841ns from start of poll, 156.775293ms from cluster start. Timeout at 2024-05-07 18:36:15.530762083 +0000 UTC m=+113.131880333, 105.536202ms after stability
2024-05-07T18:36:15.542Z [INFO]  server-c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78eea098-0044-92f9-df3f-fb4533c71060 Address:78eea098-0044-92f9-df3f-fb4533c71060} {Suffrage:Voter ID:server-c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e Address:c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e}]"
2024-05-07T18:36:15.542Z [INFO]  server-78eea098-0044-92f9-df3f-fb4533c71060: aborting pipeline replication: peer="{Voter server-c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e c1068be8-d6d8-0c03-7fe4-bfcc1a75e64e}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s)
=== RUN   TestRaft_LogStoreIsMonotonic
2024-05-07T18:36:15.543Z [INFO]  server-29295bde-9468-1b68-6a03-a3a6d044db38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29295bde-9468-1b68-6a03-a3a6d044db38 Address:29295bde-9468-1b68-6a03-a3a6d044db38}]"
2024-05-07T18:36:15.543Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:15.543Z [INFO]  server-29295bde-9468-1b68-6a03-a3a6d044db38: entering follower state: follower="Node at 29295bde-9468-1b68-6a03-a3a6d044db38 [Follower]" leader-address= leader-id=
2024-05-07T18:36:15.601Z [WARN]  server-29295bde-9468-1b68-6a03-a3a6d044db38: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:15.601Z [INFO]  server-29295bde-9468-1b68-6a03-a3a6d044db38: entering candidate state: node="Node at 29295bde-9468-1b68-6a03-a3a6d044db38 [Candidate]" term=2
2024-05-07T18:36:15.601Z [INFO]  server-29295bde-9468-1b68-6a03-a3a6d044db38: election won: term=2 tally=1
2024-05-07T18:36:15.601Z [INFO]  server-29295bde-9468-1b68-6a03-a3a6d044db38: entering leader state: leader="Node at 29295bde-9468-1b68-6a03-a3a6d044db38 [Leader]"
2024-05-07T18:36:15.707Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:15.601209351 +0000 UTC m=+113.202327601 (1 nodes), 57.712502ms from start of poll, 57.816185ms from cluster start. Timeout at 2024-05-07 18:36:15.707213232 +0000 UTC m=+113.308331482, 106.003881ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.16s)
=== RUN   TestRaft_CacheLogWithStoreError
2024-05-07T18:36:15.708Z [INFO]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bc3b66f-e523-ff81-ccad-65d790c122b6 Address:5bc3b66f-e523-ff81-ccad-65d790c122b6} {Suffrage:Voter ID:server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8 Address:b9eafbde-9b2f-eacd-8304-7984db5f4ea8}]"
2024-05-07T18:36:15.708Z [INFO]  server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bc3b66f-e523-ff81-ccad-65d790c122b6 Address:5bc3b66f-e523-ff81-ccad-65d790c122b6} {Suffrage:Voter ID:server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8 Address:b9eafbde-9b2f-eacd-8304-7984db5f4ea8}]"
2024-05-07T18:36:15.708Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:15.708Z [INFO]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: entering follower state: follower="Node at 5bc3b66f-e523-ff81-ccad-65d790c122b6 [Follower]" leader-address= leader-id=
2024-05-07T18:36:15.708Z [INFO]  server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8: entering follower state: follower="Node at b9eafbde-9b2f-eacd-8304-7984db5f4ea8 [Follower]" leader-address= leader-id=
2024-05-07T18:36:15.763Z [WARN]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:15.763Z [INFO]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: entering candidate state: node="Node at 5bc3b66f-e523-ff81-ccad-65d790c122b6 [Candidate]" term=2
2024-05-07T18:36:15.763Z [INFO]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: election won: term=2 tally=2
2024-05-07T18:36:15.763Z [INFO]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: entering leader state: leader="Node at 5bc3b66f-e523-ff81-ccad-65d790c122b6 [Leader]"
2024-05-07T18:36:15.763Z [INFO]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: added peer, starting replication: peer=server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8
2024-05-07T18:36:15.764Z [INFO]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: pipelining replication: peer="{Voter server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8 b9eafbde-9b2f-eacd-8304-7984db5f4ea8}"
2024-05-07T18:36:15.870Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:15.764119925 +0000 UTC m=+113.365238185 (1 nodes), 56.012911ms from start of poll, 56.208184ms from cluster start. Timeout at 2024-05-07 18:36:15.870128325 +0000 UTC m=+113.471246575, 106.00839ms after stability
2024-05-07T18:36:15.870Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:15.975Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:15.870203635 +0000 UTC m=+113.471321885 (1 nodes), 581ns from start of poll, 162.291884ms from cluster start. Timeout at 2024-05-07 18:36:15.975603943 +0000 UTC m=+113.576722204, 105.400319ms after stability
2024-05-07T18:36:15.976Z [ERROR] server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8: failed to append to logs: error="unable to store logs within log store, err: "some error""
2024-05-07T18:36:15.976Z [INFO]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: aborting pipeline replication: peer="{Voter server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8 b9eafbde-9b2f-eacd-8304-7984db5f4ea8}"
2024-05-07T18:36:15.976Z [WARN]  server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2024-05-07T18:36:15.982Z [INFO]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: pipelining replication: peer="{Voter server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8 b9eafbde-9b2f-eacd-8304-7984db5f4ea8}"
2024-05-07T18:36:15.982Z [INFO]  server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bc3b66f-e523-ff81-ccad-65d790c122b6 Address:5bc3b66f-e523-ff81-ccad-65d790c122b6} {Suffrage:Voter ID:server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8 Address:b9eafbde-9b2f-eacd-8304-7984db5f4ea8}]"
2024-05-07T18:36:15.982Z [INFO]  server-5bc3b66f-e523-ff81-ccad-65d790c122b6: aborting pipeline replication: peer="{Voter server-b9eafbde-9b2f-eacd-8304-7984db5f4ea8 b9eafbde-9b2f-eacd-8304-7984db5f4ea8}"
--- PASS: TestRaft_CacheLogWithStoreError (0.28s)
=== RUN   TestRaft_ReloadConfig
2024-05-07T18:36:15.984Z [INFO]  server-387ffe78-ea06-11c1-af38-9cc51f6e3f98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-387ffe78-ea06-11c1-af38-9cc51f6e3f98 Address:387ffe78-ea06-11c1-af38-9cc51f6e3f98}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2024-05-07T18:36:15.987Z [INFO]  server-39b202d8-ac98-0d70-e797-81c927bfd685: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-39b202d8-ac98-0d70-e797-81c927bfd685 Address:39b202d8-ac98-0d70-e797-81c927bfd685}]"
2024-05-07T18:36:15.987Z [INFO]  server-39b202d8-ac98-0d70-e797-81c927bfd685: entering follower state: follower="Node at 39b202d8-ac98-0d70-e797-81c927bfd685 [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2024-05-07T18:36:15.987Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2024-05-07T18:36:15.988Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53452f2a-927e-6864-0878-874a9d20b0f5 Address:53452f2a-927e-6864-0878-874a9d20b0f5} {Suffrage:Voter ID:server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9 Address:3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9} {Suffrage:Voter ID:server-8590a0fe-589e-6280-411c-e3df31fa5d30 Address:8590a0fe-589e-6280-411c-e3df31fa5d30}]"
2024-05-07T18:36:15.988Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: entering follower state: follower="Node at 53452f2a-927e-6864-0878-874a9d20b0f5 [Follower]" leader-address= leader-id=
2024-05-07T18:36:15.988Z [INFO]  server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53452f2a-927e-6864-0878-874a9d20b0f5 Address:53452f2a-927e-6864-0878-874a9d20b0f5} {Suffrage:Voter ID:server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9 Address:3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9} {Suffrage:Voter ID:server-8590a0fe-589e-6280-411c-e3df31fa5d30 Address:8590a0fe-589e-6280-411c-e3df31fa5d30}]"
2024-05-07T18:36:15.988Z [INFO]  server-8590a0fe-589e-6280-411c-e3df31fa5d30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53452f2a-927e-6864-0878-874a9d20b0f5 Address:53452f2a-927e-6864-0878-874a9d20b0f5} {Suffrage:Voter ID:server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9 Address:3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9} {Suffrage:Voter ID:server-8590a0fe-589e-6280-411c-e3df31fa5d30 Address:8590a0fe-589e-6280-411c-e3df31fa5d30}]"
2024-05-07T18:36:15.988Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:15.988Z [INFO]  server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9: entering follower state: follower="Node at 3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9 [Follower]" leader-address= leader-id=
2024-05-07T18:36:15.988Z [INFO]  server-8590a0fe-589e-6280-411c-e3df31fa5d30: entering follower state: follower="Node at 8590a0fe-589e-6280-411c-e3df31fa5d30 [Follower]" leader-address= leader-id=
2024-05-07T18:36:16.043Z [WARN]  server-53452f2a-927e-6864-0878-874a9d20b0f5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:16.043Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: entering candidate state: node="Node at 53452f2a-927e-6864-0878-874a9d20b0f5 [Candidate]" term=2
2024-05-07T18:36:16.043Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: election won: term=2 tally=2
2024-05-07T18:36:16.043Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: entering leader state: leader="Node at 53452f2a-927e-6864-0878-874a9d20b0f5 [Leader]"
2024-05-07T18:36:16.044Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: added peer, starting replication: peer=server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9
2024-05-07T18:36:16.044Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: added peer, starting replication: peer=server-8590a0fe-589e-6280-411c-e3df31fa5d30
2024-05-07T18:36:16.044Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: pipelining replication: peer="{Voter server-8590a0fe-589e-6280-411c-e3df31fa5d30 8590a0fe-589e-6280-411c-e3df31fa5d30}"
2024-05-07T18:36:16.044Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: pipelining replication: peer="{Voter server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9 3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9}"
2024-05-07T18:36:16.044Z [WARN]  server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9: rejecting vote request since our last term is greater: candidate=53452f2a-927e-6864-0878-874a9d20b0f5 last-term=2 last-candidate-term=1
2024-05-07T18:36:16.149Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:16.044207171 +0000 UTC m=+113.645325432 (1 nodes), 55.543438ms from start of poll, 55.760482ms from cluster start. Timeout at 2024-05-07 18:36:16.149610286 +0000 UTC m=+113.750728546, 105.403114ms after stability
2024-05-07T18:36:16.155Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:16.260Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:16.155406086 +0000 UTC m=+113.756524346 (2 nodes), 2.926µs from start of poll, 166.959396ms from cluster start. Timeout at 2024-05-07 18:36:16.260794241 +0000 UTC m=+113.861912491, 105.388145ms after stability
2024-05-07T18:36:16.260Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: updating configuration: command=RemoveServer server-id=server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9 server-addr= servers="[{Suffrage:Voter ID:server-53452f2a-927e-6864-0878-874a9d20b0f5 Address:53452f2a-927e-6864-0878-874a9d20b0f5} {Suffrage:Voter ID:server-8590a0fe-589e-6280-411c-e3df31fa5d30 Address:8590a0fe-589e-6280-411c-e3df31fa5d30}]"
2024-05-07T18:36:16.260Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: removed peer, stopping replication: peer=server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9 last-index=3
2024-05-07T18:36:16.260Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: aborting pipeline replication: peer="{Voter server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9 3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9}"
2024-05-07T18:36:16.365Z [WARN]  server-3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2024-05-07T18:36:16.369Z [ERROR] server-53452f2a-927e-6864-0878-874a9d20b0f5: failed to pipeline appendEntries: peer="{Voter server-8590a0fe-589e-6280-411c-e3df31fa5d30 8590a0fe-589e-6280-411c-e3df31fa5d30}" error="append pipeline closed"
2024-05-07T18:36:16.369Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: aborting pipeline replication: peer="{Voter server-8590a0fe-589e-6280-411c-e3df31fa5d30 8590a0fe-589e-6280-411c-e3df31fa5d30}"
2024-05-07T18:36:16.371Z [ERROR] server-53452f2a-927e-6864-0878-874a9d20b0f5: failed to heartbeat to: peer=8590a0fe-589e-6280-411c-e3df31fa5d30 backoff time=10ms error="failed to connect to peer: 8590a0fe-589e-6280-411c-e3df31fa5d30"
2024-05-07T18:36:16.378Z [ERROR] server-53452f2a-927e-6864-0878-874a9d20b0f5: failed to appendEntries to: peer="{Voter server-8590a0fe-589e-6280-411c-e3df31fa5d30 8590a0fe-589e-6280-411c-e3df31fa5d30}" error="failed to connect to peer: 8590a0fe-589e-6280-411c-e3df31fa5d30"
2024-05-07T18:36:16.387Z [ERROR] server-53452f2a-927e-6864-0878-874a9d20b0f5: failed to heartbeat to: peer=8590a0fe-589e-6280-411c-e3df31fa5d30 backoff time=10ms error="failed to connect to peer: 8590a0fe-589e-6280-411c-e3df31fa5d30"
2024-05-07T18:36:16.396Z [ERROR] server-53452f2a-927e-6864-0878-874a9d20b0f5: failed to appendEntries to: peer="{Voter server-8590a0fe-589e-6280-411c-e3df31fa5d30 8590a0fe-589e-6280-411c-e3df31fa5d30}" error="failed to connect to peer: 8590a0fe-589e-6280-411c-e3df31fa5d30"
2024-05-07T18:36:16.406Z [ERROR] server-53452f2a-927e-6864-0878-874a9d20b0f5: failed to heartbeat to: peer=8590a0fe-589e-6280-411c-e3df31fa5d30 backoff time=10ms error="failed to connect to peer: 8590a0fe-589e-6280-411c-e3df31fa5d30"
2024-05-07T18:36:16.414Z [ERROR] server-53452f2a-927e-6864-0878-874a9d20b0f5: failed to appendEntries to: peer="{Voter server-8590a0fe-589e-6280-411c-e3df31fa5d30 8590a0fe-589e-6280-411c-e3df31fa5d30}" error="failed to connect to peer: 8590a0fe-589e-6280-411c-e3df31fa5d30"
2024-05-07T18:36:16.415Z [WARN]  server-53452f2a-927e-6864-0878-874a9d20b0f5: failed to contact: server-id=server-8590a0fe-589e-6280-411c-e3df31fa5d30 time=50.044841ms
2024-05-07T18:36:16.415Z [WARN]  server-53452f2a-927e-6864-0878-874a9d20b0f5: failed to contact quorum of nodes, stepping down
2024-05-07T18:36:16.415Z [INFO]  server-53452f2a-927e-6864-0878-874a9d20b0f5: entering follower state: follower="Node at 53452f2a-927e-6864-0878-874a9d20b0f5 [Follower]" leader-address= leader-id=
2024-05-07T18:36:16.458Z [WARN]  server-8590a0fe-589e-6280-411c-e3df31fa5d30: heartbeat timeout reached, starting election: last-leader-addr=53452f2a-927e-6864-0878-874a9d20b0f5 last-leader-id=server-53452f2a-927e-6864-0878-874a9d20b0f5
2024-05-07T18:36:16.458Z [INFO]  server-8590a0fe-589e-6280-411c-e3df31fa5d30: entering candidate state: node="Node at 8590a0fe-589e-6280-411c-e3df31fa5d30 [Candidate]" term=3
2024-05-07T18:36:16.458Z [ERROR] server-8590a0fe-589e-6280-411c-e3df31fa5d30: failed to make requestVote RPC: target="{Voter server-53452f2a-927e-6864-0878-874a9d20b0f5 53452f2a-927e-6864-0878-874a9d20b0f5}" error="failed to connect to peer: 53452f2a-927e-6864-0878-874a9d20b0f5" term=3
2024-05-07T18:36:16.472Z [WARN]  server-8590a0fe-589e-6280-411c-e3df31fa5d30: rejecting vote request since node is not in configuration: from=3dc3f746-b7b2-4ea6-cf83-8d3079bc4ec9
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.48s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2024-05-07T18:36:16.473Z [INFO]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcba154b-68c9-1eec-b098-83d3c29d6f86 Address:fcba154b-68c9-1eec-b098-83d3c29d6f86} {Suffrage:Voter ID:server-b383600c-3aea-2ea3-e02e-2ee86275e9a4 Address:b383600c-3aea-2ea3-e02e-2ee86275e9a4} {Suffrage:Voter ID:server-8cf62be3-bc39-197e-36d7-e7ba64d0632d Address:8cf62be3-bc39-197e-36d7-e7ba64d0632d}]"
2024-05-07T18:36:16.473Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcba154b-68c9-1eec-b098-83d3c29d6f86 Address:fcba154b-68c9-1eec-b098-83d3c29d6f86} {Suffrage:Voter ID:server-b383600c-3aea-2ea3-e02e-2ee86275e9a4 Address:b383600c-3aea-2ea3-e02e-2ee86275e9a4} {Suffrage:Voter ID:server-8cf62be3-bc39-197e-36d7-e7ba64d0632d Address:8cf62be3-bc39-197e-36d7-e7ba64d0632d}]"
2024-05-07T18:36:16.473Z [INFO]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: entering follower state: follower="Node at fcba154b-68c9-1eec-b098-83d3c29d6f86 [Follower]" leader-address= leader-id=
2024-05-07T18:36:16.473Z [INFO]  server-8cf62be3-bc39-197e-36d7-e7ba64d0632d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcba154b-68c9-1eec-b098-83d3c29d6f86 Address:fcba154b-68c9-1eec-b098-83d3c29d6f86} {Suffrage:Voter ID:server-b383600c-3aea-2ea3-e02e-2ee86275e9a4 Address:b383600c-3aea-2ea3-e02e-2ee86275e9a4} {Suffrage:Voter ID:server-8cf62be3-bc39-197e-36d7-e7ba64d0632d Address:8cf62be3-bc39-197e-36d7-e7ba64d0632d}]"
2024-05-07T18:36:16.473Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:16.473Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: entering follower state: follower="Node at b383600c-3aea-2ea3-e02e-2ee86275e9a4 [Follower]" leader-address= leader-id=
2024-05-07T18:36:16.474Z [INFO]  server-8cf62be3-bc39-197e-36d7-e7ba64d0632d: entering follower state: follower="Node at 8cf62be3-bc39-197e-36d7-e7ba64d0632d [Follower]" leader-address= leader-id=
2024-05-07T18:36:16.529Z [WARN]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:16.529Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: entering candidate state: node="Node at b383600c-3aea-2ea3-e02e-2ee86275e9a4 [Candidate]" term=2
2024-05-07T18:36:16.529Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: election won: term=2 tally=2
2024-05-07T18:36:16.529Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: entering leader state: leader="Node at b383600c-3aea-2ea3-e02e-2ee86275e9a4 [Leader]"
2024-05-07T18:36:16.529Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: added peer, starting replication: peer=server-fcba154b-68c9-1eec-b098-83d3c29d6f86
2024-05-07T18:36:16.529Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: added peer, starting replication: peer=server-8cf62be3-bc39-197e-36d7-e7ba64d0632d
2024-05-07T18:36:16.529Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: pipelining replication: peer="{Voter server-8cf62be3-bc39-197e-36d7-e7ba64d0632d 8cf62be3-bc39-197e-36d7-e7ba64d0632d}"
2024-05-07T18:36:16.529Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: pipelining replication: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}"
2024-05-07T18:36:16.634Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:16.529440629 +0000 UTC m=+114.130558879 (1 nodes), 55.44742ms from start of poll, 55.784937ms from cluster start. Timeout at 2024-05-07 18:36:16.634860253 +0000 UTC m=+114.235978514, 105.419635ms after stability
2024-05-07T18:36:16.639Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:16.745Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:16.640007917 +0000 UTC m=+114.241126178 (2 nodes), 3.307µs from start of poll, 166.352236ms from cluster start. Timeout at 2024-05-07 18:36:16.745791249 +0000 UTC m=+114.346909499, 105.783321ms after stability
2024-05-07T18:36:16.746Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to heartbeat to: peer=fcba154b-68c9-1eec-b098-83d3c29d6f86 backoff time=10ms error="failed to connect to peer: fcba154b-68c9-1eec-b098-83d3c29d6f86"
2024-05-07T18:36:16.749Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to pipeline appendEntries: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}" error="append pipeline closed"
2024-05-07T18:36:16.749Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: aborting pipeline replication: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}"
2024-05-07T18:36:16.759Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to appendEntries to: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}" error="failed to connect to peer: fcba154b-68c9-1eec-b098-83d3c29d6f86"
2024-05-07T18:36:16.766Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to heartbeat to: peer=fcba154b-68c9-1eec-b098-83d3c29d6f86 backoff time=10ms error="failed to connect to peer: fcba154b-68c9-1eec-b098-83d3c29d6f86"
2024-05-07T18:36:16.776Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to appendEntries to: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}" error="failed to connect to peer: fcba154b-68c9-1eec-b098-83d3c29d6f86"
2024-05-07T18:36:16.787Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to heartbeat to: peer=fcba154b-68c9-1eec-b098-83d3c29d6f86 backoff time=10ms error="failed to connect to peer: fcba154b-68c9-1eec-b098-83d3c29d6f86"
2024-05-07T18:36:16.788Z [WARN]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to contact: server-id=server-fcba154b-68c9-1eec-b098-83d3c29d6f86 time=50.037948ms
2024-05-07T18:36:16.795Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to appendEntries to: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}" error="failed to connect to peer: fcba154b-68c9-1eec-b098-83d3c29d6f86"
2024-05-07T18:36:16.804Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to heartbeat to: peer=fcba154b-68c9-1eec-b098-83d3c29d6f86 backoff time=20ms error="failed to connect to peer: fcba154b-68c9-1eec-b098-83d3c29d6f86"
2024-05-07T18:36:16.822Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to appendEntries to: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}" error="failed to connect to peer: fcba154b-68c9-1eec-b098-83d3c29d6f86"
2024-05-07T18:36:16.826Z [WARN]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: heartbeat timeout reached, starting election: last-leader-addr=b383600c-3aea-2ea3-e02e-2ee86275e9a4 last-leader-id=server-b383600c-3aea-2ea3-e02e-2ee86275e9a4
2024-05-07T18:36:16.827Z [INFO]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: entering candidate state: node="Node at fcba154b-68c9-1eec-b098-83d3c29d6f86 [Candidate]" term=3
2024-05-07T18:36:16.827Z [ERROR] server-fcba154b-68c9-1eec-b098-83d3c29d6f86: failed to make requestVote RPC: target="{Voter server-8cf62be3-bc39-197e-36d7-e7ba64d0632d 8cf62be3-bc39-197e-36d7-e7ba64d0632d}" error="failed to connect to peer: 8cf62be3-bc39-197e-36d7-e7ba64d0632d" term=3
2024-05-07T18:36:16.827Z [ERROR] server-fcba154b-68c9-1eec-b098-83d3c29d6f86: failed to make requestVote RPC: target="{Voter server-b383600c-3aea-2ea3-e02e-2ee86275e9a4 b383600c-3aea-2ea3-e02e-2ee86275e9a4}" error="failed to connect to peer: b383600c-3aea-2ea3-e02e-2ee86275e9a4" term=3
2024-05-07T18:36:16.833Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to heartbeat to: peer=fcba154b-68c9-1eec-b098-83d3c29d6f86 backoff time=25ms error="failed to connect to peer: fcba154b-68c9-1eec-b098-83d3c29d6f86"
2024-05-07T18:36:16.839Z [WARN]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to contact: server-id=server-fcba154b-68c9-1eec-b098-83d3c29d6f86 time=100.970099ms
2024-05-07T18:36:16.850Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: updating configuration: command=RemoveServer server-id=server-fcba154b-68c9-1eec-b098-83d3c29d6f86 server-addr= servers="[{Suffrage:Voter ID:server-b383600c-3aea-2ea3-e02e-2ee86275e9a4 Address:b383600c-3aea-2ea3-e02e-2ee86275e9a4} {Suffrage:Voter ID:server-8cf62be3-bc39-197e-36d7-e7ba64d0632d Address:8cf62be3-bc39-197e-36d7-e7ba64d0632d}]"
2024-05-07T18:36:16.851Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: removed peer, stopping replication: peer=server-fcba154b-68c9-1eec-b098-83d3c29d6f86 last-index=3
2024-05-07T18:36:16.851Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: updating configuration: command=AddNonvoter server-id=server-fcba154b-68c9-1eec-b098-83d3c29d6f86 server-addr=fcba154b-68c9-1eec-b098-83d3c29d6f86 servers="[{Suffrage:Voter ID:server-b383600c-3aea-2ea3-e02e-2ee86275e9a4 Address:b383600c-3aea-2ea3-e02e-2ee86275e9a4} {Suffrage:Voter ID:server-8cf62be3-bc39-197e-36d7-e7ba64d0632d Address:8cf62be3-bc39-197e-36d7-e7ba64d0632d} {Suffrage:Nonvoter ID:server-fcba154b-68c9-1eec-b098-83d3c29d6f86 Address:fcba154b-68c9-1eec-b098-83d3c29d6f86}]"
2024-05-07T18:36:16.851Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: added peer, starting replication: peer=server-fcba154b-68c9-1eec-b098-83d3c29d6f86
2024-05-07T18:36:16.851Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: failed to appendEntries to: peer="{Nonvoter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}" error="failed to connect to peer: fcba154b-68c9-1eec-b098-83d3c29d6f86"
2024-05-07T18:36:16.868Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: peer has newer term, stopping replication: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}"
2024-05-07T18:36:16.868Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: entering follower state: follower="Node at b383600c-3aea-2ea3-e02e-2ee86275e9a4 [Follower]" leader-address= leader-id=
2024-05-07T18:36:16.868Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: aborting pipeline replication: peer="{Voter server-8cf62be3-bc39-197e-36d7-e7ba64d0632d 8cf62be3-bc39-197e-36d7-e7ba64d0632d}"
2024-05-07T18:36:16.871Z [ERROR] server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: peer has newer term, stopping replication: peer="{Nonvoter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}"
2024-05-07T18:36:16.880Z [WARN]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: Election timeout reached, restarting election
2024-05-07T18:36:16.880Z [INFO]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: entering candidate state: node="Node at fcba154b-68c9-1eec-b098-83d3c29d6f86 [Candidate]" term=23
2024-05-07T18:36:16.880Z [WARN]  server-8cf62be3-bc39-197e-36d7-e7ba64d0632d: rejecting vote request since we have a leader: from=fcba154b-68c9-1eec-b098-83d3c29d6f86 leader=b383600c-3aea-2ea3-e02e-2ee86275e9a4 leader-id=server-b383600c-3aea-2ea3-e02e-2ee86275e9a4
2024-05-07T18:36:16.880Z [WARN]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: rejecting vote request since node is not a voter: from=fcba154b-68c9-1eec-b098-83d3c29d6f86
2024-05-07T18:36:16.939Z [WARN]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: Election timeout reached, restarting election
2024-05-07T18:36:16.939Z [INFO]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: entering candidate state: node="Node at fcba154b-68c9-1eec-b098-83d3c29d6f86 [Candidate]" term=24
2024-05-07T18:36:16.939Z [WARN]  server-8cf62be3-bc39-197e-36d7-e7ba64d0632d: rejecting vote request since we have a leader: from=fcba154b-68c9-1eec-b098-83d3c29d6f86 leader=b383600c-3aea-2ea3-e02e-2ee86275e9a4 leader-id=server-b383600c-3aea-2ea3-e02e-2ee86275e9a4
2024-05-07T18:36:16.939Z [WARN]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: rejecting vote request since node is not a voter: from=fcba154b-68c9-1eec-b098-83d3c29d6f86
2024-05-07T18:36:16.946Z [WARN]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:16.946Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: entering candidate state: node="Node at b383600c-3aea-2ea3-e02e-2ee86275e9a4 [Candidate]" term=25
2024-05-07T18:36:16.946Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: election won: term=25 tally=2
2024-05-07T18:36:16.946Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: entering leader state: leader="Node at b383600c-3aea-2ea3-e02e-2ee86275e9a4 [Leader]"
2024-05-07T18:36:16.946Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: added peer, starting replication: peer=server-8cf62be3-bc39-197e-36d7-e7ba64d0632d
2024-05-07T18:36:16.946Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: added peer, starting replication: peer=server-fcba154b-68c9-1eec-b098-83d3c29d6f86
2024-05-07T18:36:16.946Z [WARN]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2024-05-07T18:36:16.946Z [INFO]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: entering follower state: follower="Node at fcba154b-68c9-1eec-b098-83d3c29d6f86 [Follower]" leader-address=b383600c-3aea-2ea3-e02e-2ee86275e9a4 leader-id=server-b383600c-3aea-2ea3-e02e-2ee86275e9a4
2024-05-07T18:36:16.946Z [WARN]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: appendEntries rejected, sending older logs: peer="{Nonvoter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}" next=3
2024-05-07T18:36:16.946Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: pipelining replication: peer="{Nonvoter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}"
2024-05-07T18:36:16.946Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: pipelining replication: peer="{Voter server-8cf62be3-bc39-197e-36d7-e7ba64d0632d 8cf62be3-bc39-197e-36d7-e7ba64d0632d}"
2024-05-07T18:36:16.957Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:17.063Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:16.957552243 +0000 UTC m=+114.558670493 (1 nodes), 86.631µs from start of poll, 483.896551ms from cluster start. Timeout at 2024-05-07 18:36:17.063080199 +0000 UTC m=+114.664198449, 105.527956ms after stability
2024-05-07T18:36:17.087Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: updating configuration: command=RemoveServer server-id=server-fcba154b-68c9-1eec-b098-83d3c29d6f86 server-addr= servers="[{Suffrage:Voter ID:server-b383600c-3aea-2ea3-e02e-2ee86275e9a4 Address:b383600c-3aea-2ea3-e02e-2ee86275e9a4} {Suffrage:Voter ID:server-8cf62be3-bc39-197e-36d7-e7ba64d0632d Address:8cf62be3-bc39-197e-36d7-e7ba64d0632d}]"
2024-05-07T18:36:17.088Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: removed peer, stopping replication: peer=server-fcba154b-68c9-1eec-b098-83d3c29d6f86 last-index=106
2024-05-07T18:36:17.088Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: aborting pipeline replication: peer="{Nonvoter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}"
2024-05-07T18:36:17.088Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: updating configuration: command=AddVoter server-id=server-fcba154b-68c9-1eec-b098-83d3c29d6f86 server-addr=fcba154b-68c9-1eec-b098-83d3c29d6f86 servers="[{Suffrage:Voter ID:server-b383600c-3aea-2ea3-e02e-2ee86275e9a4 Address:b383600c-3aea-2ea3-e02e-2ee86275e9a4} {Suffrage:Voter ID:server-8cf62be3-bc39-197e-36d7-e7ba64d0632d Address:8cf62be3-bc39-197e-36d7-e7ba64d0632d} {Suffrage:Voter ID:server-fcba154b-68c9-1eec-b098-83d3c29d6f86 Address:fcba154b-68c9-1eec-b098-83d3c29d6f86}]"
2024-05-07T18:36:17.088Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: added peer, starting replication: peer=server-fcba154b-68c9-1eec-b098-83d3c29d6f86
2024-05-07T18:36:17.088Z [WARN]  server-fcba154b-68c9-1eec-b098-83d3c29d6f86: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2024-05-07T18:36:17.088Z [WARN]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: appendEntries rejected, sending older logs: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}" next=107
2024-05-07T18:36:17.088Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: pipelining replication: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}"
2024-05-07T18:36:18.157Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:18.263Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:18.157379213 +0000 UTC m=+115.758497463 (1 nodes), 1.232µs from start of poll, 1.683723521s from cluster start. Timeout at 2024-05-07 18:36:18.263146915 +0000 UTC m=+115.864265165, 105.767702ms after stability
2024-05-07T18:36:18.263Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: aborting pipeline replication: peer="{Voter server-fcba154b-68c9-1eec-b098-83d3c29d6f86 fcba154b-68c9-1eec-b098-83d3c29d6f86}"
2024-05-07T18:36:18.263Z [INFO]  server-b383600c-3aea-2ea3-e02e-2ee86275e9a4: aborting pipeline replication: peer="{Voter server-8cf62be3-bc39-197e-36d7-e7ba64d0632d 8cf62be3-bc39-197e-36d7-e7ba64d0632d}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.79s)
=== RUN   TestRaft_FollowerRemovalNoElection
2024-05-07T18:36:18.264Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93b7df37-2e1d-025d-9626-08f04dad7a28 Address:93b7df37-2e1d-025d-9626-08f04dad7a28} {Suffrage:Voter ID:server-5b948cf4-41dd-fa2c-a9cf-408f6f413346 Address:5b948cf4-41dd-fa2c-a9cf-408f6f413346} {Suffrage:Voter ID:server-1e776f7d-6bd9-07e5-6aec-c7259d7072a5 Address:1e776f7d-6bd9-07e5-6aec-c7259d7072a5}]"
2024-05-07T18:36:18.265Z [INFO]  server-5b948cf4-41dd-fa2c-a9cf-408f6f413346: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93b7df37-2e1d-025d-9626-08f04dad7a28 Address:93b7df37-2e1d-025d-9626-08f04dad7a28} {Suffrage:Voter ID:server-5b948cf4-41dd-fa2c-a9cf-408f6f413346 Address:5b948cf4-41dd-fa2c-a9cf-408f6f413346} {Suffrage:Voter ID:server-1e776f7d-6bd9-07e5-6aec-c7259d7072a5 Address:1e776f7d-6bd9-07e5-6aec-c7259d7072a5}]"
2024-05-07T18:36:18.265Z [INFO]  server-1e776f7d-6bd9-07e5-6aec-c7259d7072a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93b7df37-2e1d-025d-9626-08f04dad7a28 Address:93b7df37-2e1d-025d-9626-08f04dad7a28} {Suffrage:Voter ID:server-5b948cf4-41dd-fa2c-a9cf-408f6f413346 Address:5b948cf4-41dd-fa2c-a9cf-408f6f413346} {Suffrage:Voter ID:server-1e776f7d-6bd9-07e5-6aec-c7259d7072a5 Address:1e776f7d-6bd9-07e5-6aec-c7259d7072a5}]"
2024-05-07T18:36:18.265Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:18.265Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: entering follower state: follower="Node at 93b7df37-2e1d-025d-9626-08f04dad7a28 [Follower]" leader-address= leader-id=
2024-05-07T18:36:18.265Z [INFO]  server-5b948cf4-41dd-fa2c-a9cf-408f6f413346: entering follower state: follower="Node at 5b948cf4-41dd-fa2c-a9cf-408f6f413346 [Follower]" leader-address= leader-id=
2024-05-07T18:36:18.265Z [INFO]  server-1e776f7d-6bd9-07e5-6aec-c7259d7072a5: entering follower state: follower="Node at 1e776f7d-6bd9-07e5-6aec-c7259d7072a5 [Follower]" leader-address= leader-id=
2024-05-07T18:36:18.390Z [WARN]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:18.390Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: entering candidate state: node="Node at 93b7df37-2e1d-025d-9626-08f04dad7a28 [Candidate]" term=2
2024-05-07T18:36:18.390Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: election won: term=2 tally=2
2024-05-07T18:36:18.390Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: entering leader state: leader="Node at 93b7df37-2e1d-025d-9626-08f04dad7a28 [Leader]"
2024-05-07T18:36:18.390Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: added peer, starting replication: peer=server-5b948cf4-41dd-fa2c-a9cf-408f6f413346
2024-05-07T18:36:18.390Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: added peer, starting replication: peer=server-1e776f7d-6bd9-07e5-6aec-c7259d7072a5
2024-05-07T18:36:18.390Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: pipelining replication: peer="{Voter server-1e776f7d-6bd9-07e5-6aec-c7259d7072a5 1e776f7d-6bd9-07e5-6aec-c7259d7072a5}"
2024-05-07T18:36:18.391Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: pipelining replication: peer="{Voter server-5b948cf4-41dd-fa2c-a9cf-408f6f413346 5b948cf4-41dd-fa2c-a9cf-408f6f413346}"
2024-05-07T18:36:18.596Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:18.390999253 +0000 UTC m=+115.992117503 (1 nodes), 125.87487ms from start of poll, 126.360583ms from cluster start. Timeout at 2024-05-07 18:36:18.596864341 +0000 UTC m=+116.197982591, 205.865088ms after stability
2024-05-07T18:36:18.596Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:18.802Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:18.596933359 +0000 UTC m=+116.198051609 (1 nodes), 791ns from start of poll, 332.294689ms from cluster start. Timeout at 2024-05-07 18:36:18.802252431 +0000 UTC m=+116.403370681, 205.319072ms after stability
2024-05-07T18:36:18.808Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:19.014Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:18.808453344 +0000 UTC m=+116.409571594 (2 nodes), 1.533µs from start of poll, 543.814674ms from cluster start. Timeout at 2024-05-07 18:36:19.014266736 +0000 UTC m=+116.615384996, 205.813402ms after stability
2024-05-07T18:36:19.014Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:19.219Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:19.014374296 +0000 UTC m=+116.615492556 (1 nodes), 2.435µs from start of poll, 749.735636ms from cluster start. Timeout at 2024-05-07 18:36:19.219538591 +0000 UTC m=+116.820656841, 205.164285ms after stability
    raft_test.go:2961: [INFO] restarting Node at 5b948cf4-41dd-fa2c-a9cf-408f6f413346 [Follower]
2024-05-07T18:36:19.219Z [INFO]  server-5b948cf4-41dd-fa2c-a9cf-408f6f413346: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93b7df37-2e1d-025d-9626-08f04dad7a28 Address:93b7df37-2e1d-025d-9626-08f04dad7a28} {Suffrage:Voter ID:server-5b948cf4-41dd-fa2c-a9cf-408f6f413346 Address:5b948cf4-41dd-fa2c-a9cf-408f6f413346} {Suffrage:Voter ID:server-1e776f7d-6bd9-07e5-6aec-c7259d7072a5 Address:1e776f7d-6bd9-07e5-6aec-c7259d7072a5}]"
2024-05-07T18:36:19.219Z [INFO]  server-5b948cf4-41dd-fa2c-a9cf-408f6f413346: entering follower state: follower="Node at 5b948cf4-41dd-fa2c-a9cf-408f6f413346 [Follower]" leader-address= leader-id=
2024-05-07T18:36:19.470Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: aborting pipeline replication: peer="{Voter server-1e776f7d-6bd9-07e5-6aec-c7259d7072a5 1e776f7d-6bd9-07e5-6aec-c7259d7072a5}"
2024-05-07T18:36:19.841Z [ERROR] server-93b7df37-2e1d-025d-9626-08f04dad7a28: failed to pipeline appendEntries: peer="{Voter server-5b948cf4-41dd-fa2c-a9cf-408f6f413346 5b948cf4-41dd-fa2c-a9cf-408f6f413346}" error="command enqueue timeout"
2024-05-07T18:36:19.841Z [INFO]  server-93b7df37-2e1d-025d-9626-08f04dad7a28: aborting pipeline replication: peer="{Voter server-5b948cf4-41dd-fa2c-a9cf-408f6f413346 5b948cf4-41dd-fa2c-a9cf-408f6f413346}"
2024-05-07T18:36:19.973Z [ERROR] server-93b7df37-2e1d-025d-9626-08f04dad7a28: failed to heartbeat to: peer=5b948cf4-41dd-fa2c-a9cf-408f6f413346 backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.71s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2024-05-07T18:36:19.974Z [INFO]  server-59530925-2b82-69bc-fb98-875449fa3e44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59530925-2b82-69bc-fb98-875449fa3e44 Address:59530925-2b82-69bc-fb98-875449fa3e44} {Suffrage:Voter ID:server-488d5b1d-c710-4d32-fcbb-7fa23fad0686 Address:488d5b1d-c710-4d32-fcbb-7fa23fad0686} {Suffrage:Voter ID:server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a Address:a71fb8e6-3ad8-b5ca-f5ff-37692207a95a}]"
2024-05-07T18:36:19.974Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59530925-2b82-69bc-fb98-875449fa3e44 Address:59530925-2b82-69bc-fb98-875449fa3e44} {Suffrage:Voter ID:server-488d5b1d-c710-4d32-fcbb-7fa23fad0686 Address:488d5b1d-c710-4d32-fcbb-7fa23fad0686} {Suffrage:Voter ID:server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a Address:a71fb8e6-3ad8-b5ca-f5ff-37692207a95a}]"
2024-05-07T18:36:19.974Z [INFO]  server-59530925-2b82-69bc-fb98-875449fa3e44: entering follower state: follower="Node at 59530925-2b82-69bc-fb98-875449fa3e44 [Follower]" leader-address= leader-id=
2024-05-07T18:36:19.974Z [INFO]  server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59530925-2b82-69bc-fb98-875449fa3e44 Address:59530925-2b82-69bc-fb98-875449fa3e44} {Suffrage:Voter ID:server-488d5b1d-c710-4d32-fcbb-7fa23fad0686 Address:488d5b1d-c710-4d32-fcbb-7fa23fad0686} {Suffrage:Voter ID:server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a Address:a71fb8e6-3ad8-b5ca-f5ff-37692207a95a}]"
2024-05-07T18:36:19.974Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:19.974Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: entering follower state: follower="Node at 488d5b1d-c710-4d32-fcbb-7fa23fad0686 [Follower]" leader-address= leader-id=
2024-05-07T18:36:19.974Z [INFO]  server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a: entering follower state: follower="Node at a71fb8e6-3ad8-b5ca-f5ff-37692207a95a [Follower]" leader-address= leader-id=
2024-05-07T18:36:20.040Z [WARN]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-05-07T18:36:20.040Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: entering candidate state: node="Node at 488d5b1d-c710-4d32-fcbb-7fa23fad0686 [Candidate]" term=2
2024-05-07T18:36:20.040Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: election won: term=2 tally=2
2024-05-07T18:36:20.040Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: entering leader state: leader="Node at 488d5b1d-c710-4d32-fcbb-7fa23fad0686 [Leader]"
2024-05-07T18:36:20.040Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: added peer, starting replication: peer=server-59530925-2b82-69bc-fb98-875449fa3e44
2024-05-07T18:36:20.040Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: added peer, starting replication: peer=server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a
2024-05-07T18:36:20.040Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: pipelining replication: peer="{Voter server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a a71fb8e6-3ad8-b5ca-f5ff-37692207a95a}"
2024-05-07T18:36:20.040Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: pipelining replication: peer="{Voter server-59530925-2b82-69bc-fb98-875449fa3e44 59530925-2b82-69bc-fb98-875449fa3e44}"
2024-05-07T18:36:20.146Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:20.040498389 +0000 UTC m=+117.641616639 (1 nodes), 65.732338ms from start of poll, 66.042656ms from cluster start. Timeout at 2024-05-07 18:36:20.146616282 +0000 UTC m=+117.747734532, 106.117893ms after stability
2024-05-07T18:36:20.146Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-05-07T18:36:20.252Z [INFO]  cluster: stable state for Leader reached at 2024-05-07 18:36:20.146703575 +0000 UTC m=+117.747821825 (1 nodes), 1.022µs from start of poll, 172.247842ms from cluster start. Timeout at 2024-05-07 18:36:20.252269141 +0000 UTC m=+117.853387391, 105.565566ms after stability
2024-05-07T18:36:20.257Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-05-07T18:36:20.362Z [INFO]  cluster: stable state for Follower reached at 2024-05-07 18:36:20.257394283 +0000 UTC m=+117.858512543 (2 nodes), 952ns from start of poll, 282.93856ms from cluster start. Timeout at 2024-05-07 18:36:20.362694145 +0000 UTC m=+117.963812395, 105.299852ms after stability
2024-05-07T18:36:20.363Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to pipeline appendEntries: peer="{Voter server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a a71fb8e6-3ad8-b5ca-f5ff-37692207a95a}" error="append pipeline closed"
2024-05-07T18:36:20.363Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: aborting pipeline replication: peer="{Voter server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a a71fb8e6-3ad8-b5ca-f5ff-37692207a95a}"
2024-05-07T18:36:20.363Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to heartbeat to: peer=a71fb8e6-3ad8-b5ca-f5ff-37692207a95a backoff time=10ms error="failed to connect to peer: a71fb8e6-3ad8-b5ca-f5ff-37692207a95a"
2024-05-07T18:36:20.365Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to heartbeat to: peer=59530925-2b82-69bc-fb98-875449fa3e44 backoff time=10ms error="failed to connect to peer: 59530925-2b82-69bc-fb98-875449fa3e44"
2024-05-07T18:36:20.367Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to pipeline appendEntries: peer="{Voter server-59530925-2b82-69bc-fb98-875449fa3e44 59530925-2b82-69bc-fb98-875449fa3e44}" error="append pipeline closed"
2024-05-07T18:36:20.367Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: aborting pipeline replication: peer="{Voter server-59530925-2b82-69bc-fb98-875449fa3e44 59530925-2b82-69bc-fb98-875449fa3e44}"
2024-05-07T18:36:20.372Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to appendEntries to: peer="{Voter server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a a71fb8e6-3ad8-b5ca-f5ff-37692207a95a}" error="failed to connect to peer: a71fb8e6-3ad8-b5ca-f5ff-37692207a95a"
2024-05-07T18:36:20.376Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to appendEntries to: peer="{Voter server-59530925-2b82-69bc-fb98-875449fa3e44 59530925-2b82-69bc-fb98-875449fa3e44}" error="failed to connect to peer: 59530925-2b82-69bc-fb98-875449fa3e44"
2024-05-07T18:36:20.385Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to heartbeat to: peer=59530925-2b82-69bc-fb98-875449fa3e44 backoff time=10ms error="failed to connect to peer: 59530925-2b82-69bc-fb98-875449fa3e44"
2024-05-07T18:36:20.385Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to heartbeat to: peer=a71fb8e6-3ad8-b5ca-f5ff-37692207a95a backoff time=10ms error="failed to connect to peer: a71fb8e6-3ad8-b5ca-f5ff-37692207a95a"
2024-05-07T18:36:20.392Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to appendEntries to: peer="{Voter server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a a71fb8e6-3ad8-b5ca-f5ff-37692207a95a}" error="failed to connect to peer: a71fb8e6-3ad8-b5ca-f5ff-37692207a95a"
2024-05-07T18:36:20.394Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to appendEntries to: peer="{Voter server-59530925-2b82-69bc-fb98-875449fa3e44 59530925-2b82-69bc-fb98-875449fa3e44}" error="failed to connect to peer: 59530925-2b82-69bc-fb98-875449fa3e44"
2024-05-07T18:36:20.402Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to heartbeat to: peer=a71fb8e6-3ad8-b5ca-f5ff-37692207a95a backoff time=10ms error="failed to connect to peer: a71fb8e6-3ad8-b5ca-f5ff-37692207a95a"
2024-05-07T18:36:20.405Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to heartbeat to: peer=59530925-2b82-69bc-fb98-875449fa3e44 backoff time=10ms error="failed to connect to peer: 59530925-2b82-69bc-fb98-875449fa3e44"
2024-05-07T18:36:20.406Z [WARN]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to contact: server-id=server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a time=50.439334ms
2024-05-07T18:36:20.412Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to appendEntries to: peer="{Voter server-59530925-2b82-69bc-fb98-875449fa3e44 59530925-2b82-69bc-fb98-875449fa3e44}" error="failed to connect to peer: 59530925-2b82-69bc-fb98-875449fa3e44"
2024-05-07T18:36:20.413Z [ERROR] server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to appendEntries to: peer="{Voter server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a a71fb8e6-3ad8-b5ca-f5ff-37692207a95a}" error="failed to connect to peer: a71fb8e6-3ad8-b5ca-f5ff-37692207a95a"
2024-05-07T18:36:20.416Z [WARN]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to contact: server-id=server-59530925-2b82-69bc-fb98-875449fa3e44 time=56.0824ms
2024-05-07T18:36:20.416Z [WARN]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to contact: server-id=server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a time=61.052434ms
2024-05-07T18:36:20.416Z [WARN]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: failed to contact quorum of nodes, stepping down
2024-05-07T18:36:20.416Z [INFO]  server-488d5b1d-c710-4d32-fcbb-7fa23fad0686: entering follower state: follower="Node at 488d5b1d-c710-4d32-fcbb-7fa23fad0686 [Follower]" leader-address= leader-id=
2024-05-07T18:36:20.451Z [WARN]  server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a: heartbeat timeout reached, starting election: last-leader-addr=488d5b1d-c710-4d32-fcbb-7fa23fad0686 last-leader-id=server-488d5b1d-c710-4d32-fcbb-7fa23fad0686
2024-05-07T18:36:20.451Z [INFO]  server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a: entering candidate state: node="Node at a71fb8e6-3ad8-b5ca-f5ff-37692207a95a [Candidate]" term=3
2024-05-07T18:36:20.451Z [ERROR] server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a: failed to make requestVote RPC: target="{Voter server-488d5b1d-c710-4d32-fcbb-7fa23fad0686 488d5b1d-c710-4d32-fcbb-7fa23fad0686}" error="failed to connect to peer: 488d5b1d-c710-4d32-fcbb-7fa23fad0686" term=3
2024-05-07T18:36:20.451Z [WARN]  server-59530925-2b82-69bc-fb98-875449fa3e44: rejecting vote request since we have a leader: from=a71fb8e6-3ad8-b5ca-f5ff-37692207a95a leader=488d5b1d-c710-4d32-fcbb-7fa23fad0686 leader-id=server-488d5b1d-c710-4d32-fcbb-7fa23fad0686
2024-05-07T18:36:20.480Z [WARN]  server-59530925-2b82-69bc-fb98-875449fa3e44: heartbeat timeout reached, starting election: last-leader-addr=488d5b1d-c710-4d32-fcbb-7fa23fad0686 last-leader-id=server-488d5b1d-c710-4d32-fcbb-7fa23fad0686
2024-05-07T18:36:20.480Z [INFO]  server-59530925-2b82-69bc-fb98-875449fa3e44: entering candidate state: node="Node at 59530925-2b82-69bc-fb98-875449fa3e44 [Candidate]" term=3
2024-05-07T18:36:20.480Z [ERROR] server-59530925-2b82-69bc-fb98-875449fa3e44: failed to make requestVote RPC: target="{Voter server-488d5b1d-c710-4d32-fcbb-7fa23fad0686 488d5b1d-c710-4d32-fcbb-7fa23fad0686}" error="failed to connect to peer: 488d5b1d-c710-4d32-fcbb-7fa23fad0686" term=3
2024-05-07T18:36:20.480Z [INFO]  server-a71fb8e6-3ad8-b5ca-f5ff-37692207a95a: duplicate requestVote for same term: term=3
2024-05-07T18:36:20.481Z [INFO]  server-59530925-2b82-69bc-fb98-875449fa3e44: entering follower state: follower="Node at 59530925-2b82-69bc-fb98-875449fa3e44 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.51s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2024-05-07T18:36:20.482Z [INFO]  127.0.0.1:44705 :: starting node: addr=127.0.0.1:44705
2024-05-07T18:36:20.482Z [INFO]  127.0.0.1:44705 :: initial configuration: index=0 servers=[]
2024-05-07T18:36:20.482Z [INFO]  127.0.0.1:44705 :: entering follower state: follower="Node at 127.0.0.1:44705 [Follower]" leader-address= leader-id=
2024-05-07T18:36:20.547Z [WARN]  127.0.0.1:44705 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2024-05-07T18:36:20.633Z [INFO]  127.0.0.1:36465 :: starting node: addr=127.0.0.1:36465
2024-05-07T18:36:20.633Z [INFO]  127.0.0.1:36465 :: initial configuration: index=0 servers=[]
2024-05-07T18:36:20.633Z [INFO]  127.0.0.1:36465 :: entering follower state: follower="Node at 127.0.0.1:36465 [Follower]" leader-address= leader-id=
2024-05-07T18:36:20.727Z [WARN]  127.0.0.1:36465 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2024-05-07T18:36:20.785Z [INFO]  127.0.0.1:41571 :: starting node: addr=127.0.0.1:41571
2024-05-07T18:36:20.785Z [INFO]  127.0.0.1:41571 :: initial configuration: index=0 servers=[]
2024-05-07T18:36:20.785Z [INFO]  127.0.0.1:41571 :: entering follower state: follower="Node at 127.0.0.1:41571 [Follower]" leader-address= leader-id=
2024-05-07T18:36:20.839Z [WARN]  127.0.0.1:41571 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.45s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2024-05-07T18:36:20.936Z [INFO]  127.0.0.1:44379 :: starting node: addr=127.0.0.1:44379
2024-05-07T18:36:20.936Z [INFO]  127.0.0.1:44379 :: initial configuration: index=0 servers=[]
2024-05-07T18:36:20.936Z [INFO]  127.0.0.1:44379 :: entering follower state: follower="Node at 127.0.0.1:44379 [Follower]" leader-address= leader-id=
2024-05-07T18:36:20.936Z [WARN]  127.0.0.1:44379 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	118.734s
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: user script /srv/workspace/pbuilder/2764266/tmp/hooks/B01_cleanup starting
I: user script /srv/workspace/pbuilder/2764266/tmp/hooks/B01_cleanup finished
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/2764266 and its subdirectories
I: Current time: Wed May  8 08:36:32 +14 2024
I: pbuilder-time-stamp: 1715106992