Thu May 23 22:48:35 UTC 2024  I: starting to build golang-github-hashicorp-raft/unstable/arm64 on jenkins on '2024-05-23 22:48'
Thu May 23 22:48:35 UTC 2024  I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/arm64_13/20031/console.log
Thu May 23 22:48:35 UTC 2024  I: Downloading source for unstable/golang-github-hashicorp-raft=1.5.0-1
--2024-05-23 22:48:35--  http://deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.5.0-1.dsc
Connecting to 46.16.76.132:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2480 (2.4K) [text/prs.lines.tag]
Saving to: ‘golang-github-hashicorp-raft_1.5.0-1.dsc’

     0K ..                                                    100%  337M=0s

2024-05-23 22:48:35 (337 MB/s) - ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ saved [2480/2480]

Thu May 23 22:48:35 UTC 2024  I: golang-github-hashicorp-raft_1.5.0-1.dsc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 3.0 (quilt)
Source: golang-github-hashicorp-raft
Binary: golang-github-hashicorp-raft-dev
Architecture: all
Version: 1.5.0-1
Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
Uploaders: Tim Potter <tpot@hpe.com>, Dmitry Smirnov <onlyjob@debian.org>,
Homepage: https://github.com/hashicorp/raft
Standards-Version: 4.6.2
Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft
Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git
Testsuite: autopkgtest-pkg-go
Build-Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
Package-List:
 golang-github-hashicorp-raft-dev deb golang optional arch=all
Checksums-Sha1:
 cea1c93d584c6f8dfbf53aef3438ab18ec7e7222 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 015fac87182a135b38ce31d1eb99f68ca251ae23 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Checksums-Sha256:
 f4dc9aabc7784a1b9b5da36b85a3797ac37911f2a6dc54a6fd6cb60a6fd643e9 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 0d95ee03abc0b447cfb92e400d914ecee5113fd181a4ea1d51fa5b6c37288658 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Files:
 7ebc703c8f7e2d7fc02568216f2e3b0a 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 1f0ee65ab76bca0e05a75b8f9ad32d8d 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Go-Import-Path: github.com/hashicorp/raft

-----BEGIN PGP SIGNATURE-----

iQJKBAEBCgA0FiEEMD9oek78sa58GjWjtwAXP7uAWikFAmTCiiIWHGRzd2FyYnJp
Y2tAZGViaWFuLm9yZwAKCRC3ABc/u4BaKYS6D/wN81mTx6pt43be/9H7Ovy6z+5p
nv0+kgrOLtH5y4KUjNLZJuPljAp+uoELmotxp1wqkoLSYs1AwCZ11X4NVs8YoLXj
sLd+Ub+QcWb0Gejqjw7Nk9VZqqXjwfFJW54n8NC2Zy8D6OelCl47sDPmiu03v3uz
oRu4dp2DW/uKkC0vb4RczT81GBmJ19Tu2ZdDvtk7kwnvF48cqDhXMhUTQAEjoyVU
Q8kfY13blza+uXZk6xhWyYsc+wpTh6RHZIWsbpgejKmEYnZCoCdaR0UqOoau33Bs
T1OXVNM0P6vrmeXyoiEy66Q2XyHeei+HYRUgED16Ujg0qxPxSEVE5timSwcXHa/u
xb5ZcKQwqEoJISjWTFDG8gww8MBPOKNGW5BKRdQSdtDhNEsnmn58jl1lsRs5L6Xg
+DrufjU6KKqqYyOBS+H+s68KXleg2t/qP+K8GZjJl/eX3+Tw5xOdt3ZWDMWxB2qF
gS+ASmxYV0Uu/pd0MLWXjAFcllnl93MzPVkF14A8eYrghujLjN3+oUOD0RQzxpAb
LHc5b0iPnN4/YQtJX3fZi2y6c62BCZ5Slb5sB0bsgNPbk1T3qQrtMPWJXUfr32ys
VOow6eNA+VDxNxHZ3aSUiVC0lcUm6dJDYqcfk/PgDdBR/E7iigkTi3O/2rgW0Y7Z
14FQMVvbK31veGXJjA==
=okTI
-----END PGP SIGNATURE-----
Thu May 23 22:48:35 UTC 2024  I: Checking whether the package is not for us
Thu May 23 22:48:35 UTC 2024  I: Starting 1st build on remote node codethink01-arm64.debian.net.
Thu May 23 22:48:35 UTC 2024  I: Preparing to do remote build '1' on codethink01-arm64.debian.net.
Thu May 23 22:51:53 UTC 2024  I: Deleting $TMPDIR on codethink01-arm64.debian.net.
I: pbuilder: network access will be disabled during build
I: Current time: Wed Jun 25 17:11:37 -12 2025
I: pbuilder-time-stamp: 1750914697
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
gpgv: Signature made Thu Jul 27 15:15:46 2023 
gpgv:                using RSA key 303F687A4EFCB1AE7C1A35A3B700173FBB805A29
gpgv:                issuer "dswarbrick@debian.org"
gpgv: Can't check signature: No public key
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/1601072/tmp/hooks/D02_print_environment starting
I: set
  BUILDDIR='/build/reproducible-path'
  BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other'
  BUILDUSERNAME='pbuilder1'
  BUILD_ARCH='arm64'
  DEBIAN_FRONTEND='noninteractive'
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 '
  DISTRIBUTION='unstable'
  HOME='/root'
  HOST_ARCH='arm64'
  IFS=' 	
  '
  INVOCATION_ID='e564a907c8c04311894b177b8b5b453e'
  LANG='C'
  LANGUAGE='en_US:en'
  LC_ALL='C'
  MAIL='/var/mail/root'
  OPTIND='1'
  PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'
  PBCURRENTCOMMANDLINEOPERATION='build'
  PBUILDER_OPERATION='build'
  PBUILDER_PKGDATADIR='/usr/share/pbuilder'
  PBUILDER_PKGLIBDIR='/usr/lib/pbuilder'
  PBUILDER_SYSCONFDIR='/etc'
  PPID='1601072'
  PS1='# '
  PS2='> '
  PS4='+ '
  PWD='/'
  SHELL='/bin/bash'
  SHLVL='2'
  SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.2JasMBRn/pbuilderrc_7MRM --distribution unstable --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.2JasMBRn/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID='109'
  SUDO_UID='104'
  SUDO_USER='jenkins'
  TERM='unknown'
  TZ='/usr/share/zoneinfo/Etc/GMT+12'
  USER='root'
  _='/usr/bin/systemd-run'
  http_proxy='http://192.168.101.4:3128'
I: uname -a
  Linux codethink01-arm64 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Jun 25 14:05 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/1601072/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: arm64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19763 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.22-go{a} golang-1.22-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdata-optlist-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libparams-util-perl{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libsub-exporter-perl{a} libsub-install-perl{a} libsub-override-perl{a} libsub-prototype-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 
0 packages upgraded, 108 newly installed, 0 to remove and 0 not upgraded.
Need to get 87.0 MB of archives. After unpacking 628 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian unstable/main arm64 sensible-utils all 0.0.22 [22.4 kB]
Get: 2 http://deb.debian.org/debian unstable/main arm64 libmagic-mgc arm64 1:5.45-3 [314 kB]
Get: 3 http://deb.debian.org/debian unstable/main arm64 libmagic1t64 arm64 1:5.45-3 [100 kB]
Get: 4 http://deb.debian.org/debian unstable/main arm64 file arm64 1:5.45-3 [43.0 kB]
Get: 5 http://deb.debian.org/debian unstable/main arm64 gettext-base arm64 0.21-14+b1 [160 kB]
Get: 6 http://deb.debian.org/debian unstable/main arm64 libuchardet0 arm64 0.0.8-1+b1 [69.0 kB]
Get: 7 http://deb.debian.org/debian unstable/main arm64 groff-base arm64 1.23.0-4 [1130 kB]
Get: 8 http://deb.debian.org/debian unstable/main arm64 bsdextrautils arm64 2.40.1-1 [93.2 kB]
Get: 9 http://deb.debian.org/debian unstable/main arm64 libpipeline1 arm64 1.5.7-2 [36.5 kB]
Get: 10 http://deb.debian.org/debian unstable/main arm64 man-db arm64 2.12.1-1 [1394 kB]
Get: 11 http://deb.debian.org/debian unstable/main arm64 m4 arm64 1.4.19-4 [277 kB]
Get: 12 http://deb.debian.org/debian unstable/main arm64 autoconf all 2.71-3 [332 kB]
Get: 13 http://deb.debian.org/debian unstable/main arm64 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian unstable/main arm64 automake all 1:1.16.5-1.3 [823 kB]
Get: 15 http://deb.debian.org/debian unstable/main arm64 autopoint all 0.21-14 [496 kB]
Get: 16 http://deb.debian.org/debian unstable/main arm64 libdebhelper-perl all 13.15.3 [88.0 kB]
Get: 17 http://deb.debian.org/debian unstable/main arm64 libtool all 2.4.7-7 [517 kB]
Get: 18 http://deb.debian.org/debian unstable/main arm64 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian unstable/main arm64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian unstable/main arm64 libparams-util-perl arm64 1.102-3 [23.4 kB]
Get: 21 http://deb.debian.org/debian unstable/main arm64 libsub-install-perl all 0.929-1 [10.5 kB]
Get: 22 http://deb.debian.org/debian unstable/main arm64 libdata-optlist-perl all 0.114-1 [10.6 kB]
Get: 23 http://deb.debian.org/debian unstable/main arm64 libsub-exporter-perl all 0.990-1 [50.6 kB]
Get: 24 http://deb.debian.org/debian unstable/main arm64 libsub-prototype-perl arm64 0.03-2+b2 [9776 B]
Get: 25 http://deb.debian.org/debian unstable/main arm64 libsub-override-perl all 0.11-1 [10.4 kB]
Get: 26 http://deb.debian.org/debian unstable/main arm64 libfile-stripnondeterminism-perl all 1.13.1-1 [19.4 kB]
Get: 27 http://deb.debian.org/debian unstable/main arm64 dh-strip-nondeterminism all 1.13.1-1 [8620 B]
Get: 28 http://deb.debian.org/debian unstable/main arm64 libelf1t64 arm64 0.191-1+b1 [187 kB]
Get: 29 http://deb.debian.org/debian unstable/main arm64 dwz arm64 0.15-1+b1 [102 kB]
Get: 30 http://deb.debian.org/debian unstable/main arm64 libicu72 arm64 72.1-4+b1 [9224 kB]
Get: 31 http://deb.debian.org/debian unstable/main arm64 libxml2 arm64 2.9.14+dfsg-1.3+b3 [624 kB]
Get: 32 http://deb.debian.org/debian unstable/main arm64 gettext arm64 0.21-14+b1 [1249 kB]
Get: 33 http://deb.debian.org/debian unstable/main arm64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 34 http://deb.debian.org/debian unstable/main arm64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 35 http://deb.debian.org/debian unstable/main arm64 debhelper all 13.15.3 [901 kB]
Get: 36 http://deb.debian.org/debian unstable/main arm64 dh-golang all 1.62 [24.7 kB]
Get: 37 http://deb.debian.org/debian unstable/main arm64 golang-1.22-src all 1.22.3-1 [18.8 MB]
Get: 38 http://deb.debian.org/debian unstable/main arm64 golang-1.22-go arm64 1.22.3-1 [21.3 MB]
Get: 39 http://deb.debian.org/debian unstable/main arm64 golang-src all 2:1.22~3 [5020 B]
Get: 40 http://deb.debian.org/debian unstable/main arm64 golang-go arm64 2:1.22~3 [44.2 kB]
Get: 41 http://deb.debian.org/debian unstable/main arm64 golang-any arm64 2:1.22~3 [5104 B]
Get: 42 http://deb.debian.org/debian unstable/main arm64 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B]
Get: 43 http://deb.debian.org/debian unstable/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 44 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 45 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-sys-dev all 0.20.0-1 [430 kB]
Get: 46 http://deb.debian.org/debian unstable/main arm64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 47 http://deb.debian.org/debian unstable/main arm64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 48 http://deb.debian.org/debian unstable/main arm64 golang-github-fatih-color-dev all 1.16.0-1 [13.8 kB]
Get: 49 http://deb.debian.org/debian unstable/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 50 http://deb.debian.org/debian unstable/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 51 http://deb.debian.org/debian unstable/main arm64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 52 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 53 http://deb.debian.org/debian unstable/main arm64 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB]
Get: 54 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 55 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 56 http://deb.debian.org/debian unstable/main arm64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 57 http://deb.debian.org/debian unstable/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 58 http://deb.debian.org/debian unstable/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 59 http://deb.debian.org/debian unstable/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 60 http://deb.debian.org/debian unstable/main arm64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 61 http://deb.debian.org/debian unstable/main arm64 golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 62 http://deb.debian.org/debian unstable/main arm64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 63 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 64 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 65 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 66 http://deb.debian.org/debian unstable/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 67 http://deb.debian.org/debian unstable/main arm64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 68 http://deb.debian.org/debian unstable/main arm64 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B]
Get: 69 http://deb.debian.org/debian unstable/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 70 http://deb.debian.org/debian unstable/main arm64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 71 http://deb.debian.org/debian unstable/main arm64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 72 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 73 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 74 http://deb.debian.org/debian unstable/main arm64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 75 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-term-dev all 0.20.0-1 [14.8 kB]
Get: 76 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-text-dev all 0.15.0-1 [4006 kB]
Get: 77 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-crypto-dev all 1:0.23.0-1 [1689 kB]
Get: 78 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-net-dev all 1:0.25.0+dfsg-1 [895 kB]
Get: 79 http://deb.debian.org/debian unstable/main arm64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 80 http://deb.debian.org/debian unstable/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 81 http://deb.debian.org/debian unstable/main arm64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 82 http://deb.debian.org/debian unstable/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 83 http://deb.debian.org/debian unstable/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 84 http://deb.debian.org/debian unstable/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 85 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 86 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 87 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-common-dev all 0.53.0-1 [114 kB]
Get: 88 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 89 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB]
Get: 90 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-client-golang-dev all 1.19.0-1 [179 kB]
Get: 91 http://deb.debian.org/debian unstable/main arm64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 92 http://deb.debian.org/debian unstable/main arm64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 93 http://deb.debian.org/debian unstable/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 94 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 95 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 96 http://deb.debian.org/debian unstable/main arm64 libsasl2-modules-db arm64 2.1.28+dfsg1-6 [20.1 kB]
Get: 97 http://deb.debian.org/debian unstable/main arm64 libsasl2-2 arm64 2.1.28+dfsg1-6 [55.3 kB]
Get: 98 http://deb.debian.org/debian unstable/main arm64 libssl-dev arm64 3.2.1-3 [3089 kB]
Get: 99 http://deb.debian.org/debian unstable/main arm64 libsasl2-dev arm64 2.1.28+dfsg1-6 [226 kB]
Get: 100 http://deb.debian.org/debian unstable/main arm64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 101 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-mod-dev all 0.17.0-1 [103 kB]
Get: 102 http://deb.debian.org/debian unstable/main arm64 golang-github-yuin-goldmark-dev all 1.7.1-1 [113 kB]
Get: 103 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-tools-dev all 1:0.20.0+ds-1 [1571 kB]
Get: 104 http://deb.debian.org/debian unstable/main arm64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 105 http://deb.debian.org/debian unstable/main arm64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 106 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 107 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 108 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 87.0 MB in 0s (196 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19763 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.22_all.deb ...
Unpacking sensible-utils (0.0.22) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3_arm64.deb ...
Unpacking libmagic-mgc (1:5.45-3) ...
Selecting previously unselected package libmagic1t64:arm64.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3_arm64.deb ...
Unpacking libmagic1t64:arm64 (1:5.45-3) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3_arm64.deb ...
Unpacking file (1:5.45-3) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.21-14+b1_arm64.deb ...
Unpacking gettext-base (0.21-14+b1) ...
Selecting previously unselected package libuchardet0:arm64.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b1_arm64.deb ...
Unpacking libuchardet0:arm64 (0.0.8-1+b1) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-4_arm64.deb ...
Unpacking groff-base (1.23.0-4) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.1-1_arm64.deb ...
Unpacking bsdextrautils (2.40.1-1) ...
Selecting previously unselected package libpipeline1:arm64.
Preparing to unpack .../008-libpipeline1_1.5.7-2_arm64.deb ...
Unpacking libpipeline1:arm64 (1.5.7-2) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.12.1-1_arm64.deb ...
Unpacking man-db (2.12.1-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-4_arm64.deb ...
Unpacking m4 (1.4.19-4) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.71-3_all.deb ...
Unpacking autoconf (2.71-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ...
Unpacking automake (1:1.16.5-1.3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.21-14_all.deb ...
Unpacking autopoint (0.21-14) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.15.3_all.deb ...
Unpacking libdebhelper-perl (13.15.3) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.4.7-7_all.deb ...
Unpacking libtool (2.4.7-7) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libparams-util-perl.
Preparing to unpack .../019-libparams-util-perl_1.102-3_arm64.deb ...
Unpacking libparams-util-perl (1.102-3) ...
Selecting previously unselected package libsub-install-perl.
Preparing to unpack .../020-libsub-install-perl_0.929-1_all.deb ...
Unpacking libsub-install-perl (0.929-1) ...
Selecting previously unselected package libdata-optlist-perl.
Preparing to unpack .../021-libdata-optlist-perl_0.114-1_all.deb ...
Unpacking libdata-optlist-perl (0.114-1) ...
Selecting previously unselected package libsub-exporter-perl.
Preparing to unpack .../022-libsub-exporter-perl_0.990-1_all.deb ...
Unpacking libsub-exporter-perl (0.990-1) ...
Selecting previously unselected package libsub-prototype-perl.
Preparing to unpack .../023-libsub-prototype-perl_0.03-2+b2_arm64.deb ...
Unpacking libsub-prototype-perl (0.03-2+b2) ...
Selecting previously unselected package libsub-override-perl.
Preparing to unpack .../024-libsub-override-perl_0.11-1_all.deb ...
Unpacking libsub-override-perl (0.11-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../025-libfile-stripnondeterminism-perl_1.13.1-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.13.1-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../026-dh-strip-nondeterminism_1.13.1-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.13.1-1) ...
Selecting previously unselected package libelf1t64:arm64.
Preparing to unpack .../027-libelf1t64_0.191-1+b1_arm64.deb ...
Unpacking libelf1t64:arm64 (0.191-1+b1) ...
Selecting previously unselected package dwz.
Preparing to unpack .../028-dwz_0.15-1+b1_arm64.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libicu72:arm64.
Preparing to unpack .../029-libicu72_72.1-4+b1_arm64.deb ...
Unpacking libicu72:arm64 (72.1-4+b1) ...
Selecting previously unselected package libxml2:arm64.
Preparing to unpack .../030-libxml2_2.9.14+dfsg-1.3+b3_arm64.deb ...
Unpacking libxml2:arm64 (2.9.14+dfsg-1.3+b3) ...
Selecting previously unselected package gettext.
Preparing to unpack .../031-gettext_0.21-14+b1_arm64.deb ...
Unpacking gettext (0.21-14+b1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../032-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../033-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../034-debhelper_13.15.3_all.deb ...
Unpacking debhelper (13.15.3) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../035-dh-golang_1.62_all.deb ...
Unpacking dh-golang (1.62) ...
Selecting previously unselected package golang-1.22-src.
Preparing to unpack .../036-golang-1.22-src_1.22.3-1_all.deb ...
Unpacking golang-1.22-src (1.22.3-1) ...
Selecting previously unselected package golang-1.22-go.
Preparing to unpack .../037-golang-1.22-go_1.22.3-1_arm64.deb ...
Unpacking golang-1.22-go (1.22.3-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../038-golang-src_2%3a1.22~3_all.deb ...
Unpacking golang-src (2:1.22~3) ...
Selecting previously unselected package golang-go:arm64.
Preparing to unpack .../039-golang-go_2%3a1.22~3_arm64.deb ...
Unpacking golang-go:arm64 (2:1.22~3) ...
Selecting previously unselected package golang-any:arm64.
Preparing to unpack .../040-golang-any_2%3a1.22~3_arm64.deb ...
Unpacking golang-any:arm64 (2:1.22~3) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../041-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../042-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../043-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../044-golang-golang-x-sys-dev_0.20.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.20.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../045-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../046-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../047-golang-github-fatih-color-dev_1.16.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.16.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../048-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../049-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../050-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../051-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../052-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../053-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../054-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../055-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../056-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../057-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../058-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../059-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../060-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../061-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../062-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../063-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../064-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../065-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../066-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../067-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../068-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../069-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../070-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../071-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../072-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../073-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../074-golang-golang-x-term-dev_0.20.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.20.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../075-golang-golang-x-text-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.15.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../076-golang-golang-x-crypto-dev_1%3a0.23.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.23.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../077-golang-golang-x-net-dev_1%3a0.25.0+dfsg-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.25.0+dfsg-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../078-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../079-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../080-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../081-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../082-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../083-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../084-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../085-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../086-golang-github-prometheus-common-dev_0.53.0-1_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.53.0-1) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../087-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../088-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../089-golang-github-prometheus-client-golang-dev_1.19.0-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.19.0-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../090-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../091-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../092-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../093-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../094-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:arm64.
Preparing to unpack .../095-libsasl2-modules-db_2.1.28+dfsg1-6_arm64.deb ...
Unpacking libsasl2-modules-db:arm64 (2.1.28+dfsg1-6) ...
Selecting previously unselected package libsasl2-2:arm64.
Preparing to unpack .../096-libsasl2-2_2.1.28+dfsg1-6_arm64.deb ...
Unpacking libsasl2-2:arm64 (2.1.28+dfsg1-6) ...
Selecting previously unselected package libssl-dev:arm64.
Preparing to unpack .../097-libssl-dev_3.2.1-3_arm64.deb ...
Unpacking libssl-dev:arm64 (3.2.1-3) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../098-libsasl2-dev_2.1.28+dfsg1-6_arm64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-6) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../099-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../100-golang-golang-x-mod-dev_0.17.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.17.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../101-golang-github-yuin-goldmark-dev_1.7.1-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.1-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../102-golang-golang-x-tools-dev_1%3a0.20.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.20.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../103-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../104-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../105-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../106-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../107-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up libpipeline1:arm64 (1.5.7-2) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:arm64 (72.1-4+b1) ...
Setting up bsdextrautils (2.40.1-1) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.15.3) ...
Setting up golang-golang-x-sys-dev (0.20.0-1) ...
Setting up libmagic1t64:arm64 (1:5.45-3) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.21-14+b1) ...
Setting up m4 (1.4.19-4) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.20.0-1) ...
Setting up file (1:5.45-3) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libsub-install-perl (0.929-1) ...
Setting up libelf1t64:arm64 (0.191-1+b1) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:arm64 (2.1.28+dfsg1-6) ...
Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.9.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.1-1) ...
Setting up golang-1.22-src (1.22.3-1) ...
Setting up libssl-dev:arm64 (3.2.1-3) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.17.0-1) ...
Setting up autopoint (0.21-14) ...
Setting up libparams-util-perl (1.102-3) ...
Setting up libsasl2-2:arm64 (2.1.28+dfsg1-6) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.71-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up sensible-utils (0.0.22) ...
Setting up golang-golang-x-text-dev (0.15.0-1) ...
Setting up libuchardet0:arm64 (0.0.8-1+b1) ...
Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.23.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.22~3) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:arm64 (2.9.14+dfsg-1.3+b3) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.16.5-1.3) ...
update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.21-14+b1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.4.7-7) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-1.22-go (1.22.3-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg1-6) ...
Setting up libdata-optlist-perl (0.114-1) ...
Setting up golang-golang-x-net-dev (1:0.25.0+dfsg-1) ...
Setting up groff-base (1.23.0-4) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up libsub-exporter-perl (0.990-1) ...
Setting up golang-github-fatih-color-dev (1.16.0-1) ...
Setting up golang-go:arm64 (2:1.22~3) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up libsub-prototype-perl (0.03-2+b2) ...
Setting up golang-golang-x-tools-dev (1:0.20.0+ds-1) ...
Setting up man-db (2.12.1-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-any:arm64 (2:1.22~3) ...
Setting up libsub-override-perl (0.11-1) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up libfile-stripnondeterminism-perl (1.13.1-1) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-strip-nondeterminism (1.13.1-1) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up debhelper (13.15.3) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up dh-golang (1.62) ...
Setting up golang-github-prometheus-common-dev (0.53.0-1) ...
Setting up golang-github-prometheus-client-golang-dev (1.19.0-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Processing triggers for libc-bin (2.38-11) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture arm64
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-aarch64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/coverage/rtcov
internal/unsafeheader
internal/goos
internal/godebugs
internal/race
unicode/utf8
internal/goarch
unicode
sync/atomic
internal/goexperiment
container/list
runtime/internal/syscall
internal/itoa
runtime/internal/math
internal/abi
internal/cpu
internal/chacha8rand
runtime/internal/sys
math/bits
crypto/internal/alias
crypto/subtle
crypto/internal/boring/sig
cmp
encoding
unicode/utf16
log/internal
internal/nettrace
math
slices
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
internal/bytealg
runtime/internal/atomic
runtime
internal/reflectlite
sync
internal/testlog
internal/bisect
internal/singleflight
runtime/cgo
internal/godebug
errors
sort
io
internal/oserror
math/rand
internal/intern
bytes
internal/safefilepath
vendor/golang.org/x/net/dns/dnsmessage
crypto/internal/nistec/fiat
path
github.com/hashicorp/golang-lru/simplelru
strconv
strings
syscall
hash
crypto/internal/randutil
internal/saferio
hash/crc32
hash/crc64
vendor/golang.org/x/text/transform
bufio
crypto
github.com/hashicorp/go-immutable-radix
reflect
crypto/rc4
net/netip
net/http/internal/ascii
html
regexp/syntax
regexp
time
internal/syscall/execenv
internal/syscall/unix
context
internal/poll
io/fs
encoding/binary
internal/fmtsort
embed
os
crypto/cipher
crypto/md5
encoding/base64
crypto/internal/edwards25519/field
vendor/golang.org/x/crypto/internal/poly1305
crypto/internal/nistec
golang.org/x/sys/unix
encoding/pem
crypto/internal/boring
crypto/des
vendor/golang.org/x/crypto/chacha20
crypto/aes
crypto/sha512
crypto/hmac
crypto/sha1
crypto/sha256
crypto/internal/edwards25519
vendor/golang.org/x/crypto/hkdf
fmt
path/filepath
os/signal
vendor/golang.org/x/crypto/chacha20poly1305
io/ioutil
internal/sysinfo
net
crypto/ecdh
math/big
compress/flate
encoding/json
mime
vendor/golang.org/x/net/http2/hpack
mime/quotedprintable
encoding/hex
net/url
log
vendor/golang.org/x/text/unicode/norm
net/http/internal
encoding/gob
vendor/golang.org/x/text/unicode/bidi
go/token
text/template/parse
flag
runtime/debug
runtime/trace
vendor/golang.org/x/text/secure/bidirule
compress/gzip
testing
text/template
vendor/golang.org/x/net/idna
github.com/mattn/go-isatty
github.com/mattn/go-colorable
crypto/rand
crypto/elliptic
crypto/internal/bigmod
crypto/internal/boring/bbig
crypto/dsa
encoding/asn1
github.com/fatih/color
crypto/ed25519
html/template
github.com/hashicorp/go-hclog
vendor/golang.org/x/crypto/cryptobyte
crypto/rsa
crypto/x509/pkix
crypto/ecdsa
crypto/x509
net/textproto
vendor/golang.org/x/net/http/httpproxy
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
github.com/armon/go-metrics
net/rpc
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
?   	github.com/hashicorp/raft/bench	[no test files]
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2025-06-26T05:12:51.855Z [INFO]  creating new snapshot: path=/tmp/raft1503991269/raft1904268116/snapshots/3-10-1750914771855.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s)
=== RUN   TestFileSS_CreateSnapshot
2025-06-26T05:12:51.855Z [INFO]  creating new snapshot: path=/tmp/raft602870771/snapshots/3-10-1750914771855.tmp
2025-06-26T05:12:51.855Z [WARN]  found temporary snapshot: name=3-10-1750914771855.tmp
--- PASS: TestFileSS_CreateSnapshot (0.00s)
=== RUN   TestFileSS_CancelSnapshot
2025-06-26T05:12:51.856Z [INFO]  creating new snapshot: path=/tmp/raft2569040819/snapshots/3-10-1750914771856.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2025-06-26T05:12:51.856Z [INFO]  creating new snapshot: path=/tmp/raft88503183/snapshots/3-10-1750914771856.tmp
2025-06-26T05:12:51.856Z [INFO]  creating new snapshot: path=/tmp/raft88503183/snapshots/3-11-1750914771856.tmp
2025-06-26T05:12:51.856Z [INFO]  creating new snapshot: path=/tmp/raft88503183/snapshots/3-12-1750914771856.tmp
2025-06-26T05:12:51.856Z [INFO]  reaping snapshot: path=/tmp/raft88503183/snapshots/3-10-1750914771856
2025-06-26T05:12:51.856Z [INFO]  creating new snapshot: path=/tmp/raft88503183/snapshots/3-13-1750914771856.tmp
2025-06-26T05:12:51.856Z [INFO]  reaping snapshot: path=/tmp/raft88503183/snapshots/3-11-1750914771856
2025-06-26T05:12:51.856Z [INFO]  creating new snapshot: path=/tmp/raft88503183/snapshots/3-14-1750914771856.tmp
2025-06-26T05:12:51.856Z [INFO]  reaping snapshot: path=/tmp/raft88503183/snapshots/3-12-1750914771856
--- PASS: TestFileSS_Retention (0.00s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2025-06-26T05:12:51.857Z [INFO]  creating new snapshot: path=/tmp/raft2788528951/snapshots/5-130350-1750914771857.tmp
2025-06-26T05:12:51.857Z [INFO]  creating new snapshot: path=/tmp/raft2788528951/snapshots/36-204917-1750914771857.tmp
--- PASS: TestFileSS_Ordering (0.00s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.06s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.03s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2025-06-26T05:12:52.034Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2025-06-26T05:12:52.086Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.18s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.06s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.08s)
=== RUN   TestNetworkTransport_RequestVote
2025-06-26T05:12:52.162Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.03s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.04s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:52.229Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:52.237Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:52.249Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:52.273Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:52.317Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:52.403Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:52.568Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:52.891Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:53.533Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:54.537Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-06-26T05:12:55.541Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2025-06-26T05:12:56.234Z [INFO]  server-2f7c17d8-0f8a-1b62-5489-6c2651ec03c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f7c17d8-0f8a-1b62-5489-6c2651ec03c9 Address:2f7c17d8-0f8a-1b62-5489-6c2651ec03c9}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2025-06-26T05:12:56.234Z [INFO]  server-89640e47-01c1-2a30-cfd4-319b4089ee00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89640e47-01c1-2a30-cfd4-319b4089ee00 Address:89640e47-01c1-2a30-cfd4-319b4089ee00}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2025-06-26T05:12:56.234Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: initial configuration: index=0 servers=[]
2025-06-26T05:12:56.234Z [INFO]  server-9b924aa6-db9d-b11d-cd98-a7eb92732797: initial configuration: index=0 servers=[]
2025-06-26T05:12:56.234Z [INFO]  server-5c05768c-7bad-f626-a82e-c95c4a34d14a: initial configuration: index=0 servers=[]
2025-06-26T05:12:56.234Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: entering follower state: follower="Node at 701312bb-710d-35ed-dd7b-61136dda0e63 [Follower]" leader-address= leader-id=
2025-06-26T05:12:56.234Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:12:56.234Z [INFO]  server-9b924aa6-db9d-b11d-cd98-a7eb92732797: entering follower state: follower="Node at 9b924aa6-db9d-b11d-cd98-a7eb92732797 [Follower]" leader-address= leader-id=
2025-06-26T05:12:56.234Z [INFO]  server-5c05768c-7bad-f626-a82e-c95c4a34d14a: entering follower state: follower="Node at 5c05768c-7bad-f626-a82e-c95c4a34d14a [Follower]" leader-address= leader-id=
2025-06-26T05:12:56.305Z [WARN]  server-9b924aa6-db9d-b11d-cd98-a7eb92732797: no known peers, aborting election
2025-06-26T05:12:56.317Z [WARN]  server-5c05768c-7bad-f626-a82e-c95c4a34d14a: no known peers, aborting election
2025-06-26T05:12:56.329Z [WARN]  server-701312bb-710d-35ed-dd7b-61136dda0e63: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:56.329Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: entering candidate state: node="Node at 701312bb-710d-35ed-dd7b-61136dda0e63 [Candidate]" term=2
2025-06-26T05:12:56.329Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: election won: term=2 tally=2
2025-06-26T05:12:56.329Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: entering leader state: leader="Node at 701312bb-710d-35ed-dd7b-61136dda0e63 [Leader]"
2025-06-26T05:12:56.329Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: added peer, starting replication: peer=server-9b924aa6-db9d-b11d-cd98-a7eb92732797
2025-06-26T05:12:56.329Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: added peer, starting replication: peer=server-5c05768c-7bad-f626-a82e-c95c4a34d14a
2025-06-26T05:12:56.329Z [WARN]  server-5c05768c-7bad-f626-a82e-c95c4a34d14a: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-06-26T05:12:56.329Z [WARN]  server-701312bb-710d-35ed-dd7b-61136dda0e63: appendEntries rejected, sending older logs: peer="{Voter server-5c05768c-7bad-f626-a82e-c95c4a34d14a 5c05768c-7bad-f626-a82e-c95c4a34d14a}" next=1
2025-06-26T05:12:56.329Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: pipelining replication: peer="{Voter server-5c05768c-7bad-f626-a82e-c95c4a34d14a 5c05768c-7bad-f626-a82e-c95c4a34d14a}"
2025-06-26T05:12:56.329Z [WARN]  server-9b924aa6-db9d-b11d-cd98-a7eb92732797: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-06-26T05:12:56.329Z [WARN]  server-701312bb-710d-35ed-dd7b-61136dda0e63: appendEntries rejected, sending older logs: peer="{Voter server-9b924aa6-db9d-b11d-cd98-a7eb92732797 9b924aa6-db9d-b11d-cd98-a7eb92732797}" next=1
2025-06-26T05:12:56.330Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: pipelining replication: peer="{Voter server-9b924aa6-db9d-b11d-cd98-a7eb92732797 9b924aa6-db9d-b11d-cd98-a7eb92732797}"
2025-06-26T05:12:56.437Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:12:56.330045363 +0000 UTC m=+4.479155641 (2 nodes), 95.31508ms from start of poll, 95.4542ms from cluster start. Timeout at 2025-06-26 05:12:56.437607683 +0000 UTC m=+4.586717961, 107.56232ms after stability
2025-06-26T05:12:56.437Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:56.545Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:56.437761443 +0000 UTC m=+4.586871681 (1 nodes), 520ns from start of poll, 203.17024ms from cluster start. Timeout at 2025-06-26 05:12:56.545745243 +0000 UTC m=+4.694855521, 107.98384ms after stability
2025-06-26T05:12:56.561Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: aborting pipeline replication: peer="{Voter server-9b924aa6-db9d-b11d-cd98-a7eb92732797 9b924aa6-db9d-b11d-cd98-a7eb92732797}"
2025-06-26T05:12:56.561Z [INFO]  server-701312bb-710d-35ed-dd7b-61136dda0e63: aborting pipeline replication: peer="{Voter server-5c05768c-7bad-f626-a82e-c95c4a34d14a 5c05768c-7bad-f626-a82e-c95c4a34d14a}"
--- PASS: TestRaft_LiveBootstrap (0.33s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2025-06-26T05:12:56.562Z [INFO]  server-b88ee627-b8b4-9d36-2077-1dea64ca680d: initial configuration: index=0 servers=[]
2025-06-26T05:12:56.562Z [INFO]  server-12ccac30-d566-67e4-3147-d00967598156: initial configuration: index=0 servers=[]
2025-06-26T05:12:56.562Z [INFO]  server-b88ee627-b8b4-9d36-2077-1dea64ca680d: entering follower state: follower="Node at b88ee627-b8b4-9d36-2077-1dea64ca680d [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2025-06-26T05:12:56.562Z [INFO]  server-61020da0-1eda-c51b-c2b9-56880e131e98: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2025-06-26T05:12:56.562Z [INFO]  server-a3912c67-83c5-ac1e-b30e-ca9a662beb19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3912c67-83c5-ac1e-b30e-ca9a662beb19 Address:a3912c67-83c5-ac1e-b30e-ca9a662beb19} {Suffrage:Voter ID:server-e289064c-cae2-0623-0617-1cc79076933a Address:e289064c-cae2-0623-0617-1cc79076933a} {Suffrage:Voter ID:server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82 Address:76be5bdf-1aab-d840-2219-ea5f3cb0bb82}]"
2025-06-26T05:12:56.562Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3912c67-83c5-ac1e-b30e-ca9a662beb19 Address:a3912c67-83c5-ac1e-b30e-ca9a662beb19} {Suffrage:Voter ID:server-e289064c-cae2-0623-0617-1cc79076933a Address:e289064c-cae2-0623-0617-1cc79076933a} {Suffrage:Voter ID:server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82 Address:76be5bdf-1aab-d840-2219-ea5f3cb0bb82}]"
2025-06-26T05:12:56.562Z [INFO]  server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3912c67-83c5-ac1e-b30e-ca9a662beb19 Address:a3912c67-83c5-ac1e-b30e-ca9a662beb19} {Suffrage:Voter ID:server-e289064c-cae2-0623-0617-1cc79076933a Address:e289064c-cae2-0623-0617-1cc79076933a} {Suffrage:Voter ID:server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82 Address:76be5bdf-1aab-d840-2219-ea5f3cb0bb82}]"
2025-06-26T05:12:56.562Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:56.562Z [INFO]  server-a3912c67-83c5-ac1e-b30e-ca9a662beb19: entering follower state: follower="Node at a3912c67-83c5-ac1e-b30e-ca9a662beb19 [Follower]" leader-address= leader-id=
2025-06-26T05:12:56.562Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: entering follower state: follower="Node at e289064c-cae2-0623-0617-1cc79076933a [Follower]" leader-address= leader-id=
2025-06-26T05:12:56.562Z [INFO]  server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82: entering follower state: follower="Node at 76be5bdf-1aab-d840-2219-ea5f3cb0bb82 [Follower]" leader-address= leader-id=
2025-06-26T05:12:56.653Z [WARN]  server-e289064c-cae2-0623-0617-1cc79076933a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:56.653Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: entering candidate state: node="Node at e289064c-cae2-0623-0617-1cc79076933a [Candidate]" term=2
2025-06-26T05:12:56.653Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: election won: term=2 tally=2
2025-06-26T05:12:56.653Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: entering leader state: leader="Node at e289064c-cae2-0623-0617-1cc79076933a [Leader]"
2025-06-26T05:12:56.653Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: added peer, starting replication: peer=server-a3912c67-83c5-ac1e-b30e-ca9a662beb19
2025-06-26T05:12:56.653Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: added peer, starting replication: peer=server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82
2025-06-26T05:12:56.653Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: pipelining replication: peer="{Voter server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82 76be5bdf-1aab-d840-2219-ea5f3cb0bb82}"
2025-06-26T05:12:56.653Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: pipelining replication: peer="{Voter server-a3912c67-83c5-ac1e-b30e-ca9a662beb19 a3912c67-83c5-ac1e-b30e-ca9a662beb19}"
2025-06-26T05:12:56.761Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:56.653985923 +0000 UTC m=+4.803096161 (1 nodes), 91.2828ms from start of poll, 91.4598ms from cluster start. Timeout at 2025-06-26 05:12:56.761605083 +0000 UTC m=+4.910715361, 107.6192ms after stability
2025-06-26T05:12:56.761Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: aborting pipeline replication: peer="{Voter server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82 76be5bdf-1aab-d840-2219-ea5f3cb0bb82}"
2025-06-26T05:12:56.761Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: aborting pipeline replication: peer="{Voter server-a3912c67-83c5-ac1e-b30e-ca9a662beb19 a3912c67-83c5-ac1e-b30e-ca9a662beb19}"
2025-06-26T05:12:56.761Z [INFO]  creating new snapshot: path=/tmp/raft1791675891/snapshots/2-2-1750914776761.tmp
2025-06-26T05:12:56.762Z [INFO]  server-a3912c67-83c5-ac1e-b30e-ca9a662beb19: starting restore from snapshot: id=2-2-1750914776761 last-index=2 last-term=2 size-in-bytes=1
2025-06-26T05:12:56.762Z [INFO]  server-a3912c67-83c5-ac1e-b30e-ca9a662beb19: snapshot restore progress: id=2-2-1750914776761 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-06-26T05:12:56.762Z [INFO]  server-a3912c67-83c5-ac1e-b30e-ca9a662beb19: restored from snapshot: id=2-2-1750914776761 last-index=2 last-term=2 size-in-bytes=1
2025-06-26T05:12:56.762Z [INFO]  server-a3912c67-83c5-ac1e-b30e-ca9a662beb19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3912c67-83c5-ac1e-b30e-ca9a662beb19 Address:a3912c67-83c5-ac1e-b30e-ca9a662beb19} {Suffrage:Voter ID:server-e289064c-cae2-0623-0617-1cc79076933a Address:e289064c-cae2-0623-0617-1cc79076933a} {Suffrage:Voter ID:server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82 Address:76be5bdf-1aab-d840-2219-ea5f3cb0bb82}]"
2025-06-26T05:12:56.762Z [INFO]  creating new snapshot: path=/tmp/raft3498301835/snapshots/2-2-1750914776762.tmp
2025-06-26T05:12:56.762Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: starting restore from snapshot: id=2-2-1750914776762 last-index=2 last-term=2 size-in-bytes=1
2025-06-26T05:12:56.762Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: snapshot restore progress: id=2-2-1750914776762 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-06-26T05:12:56.762Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: restored from snapshot: id=2-2-1750914776762 last-index=2 last-term=2 size-in-bytes=1
2025-06-26T05:12:56.762Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3912c67-83c5-ac1e-b30e-ca9a662beb19 Address:a3912c67-83c5-ac1e-b30e-ca9a662beb19} {Suffrage:Voter ID:server-e289064c-cae2-0623-0617-1cc79076933a Address:e289064c-cae2-0623-0617-1cc79076933a} {Suffrage:Voter ID:server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82 Address:76be5bdf-1aab-d840-2219-ea5f3cb0bb82}]"
2025-06-26T05:12:56.762Z [INFO]  creating new snapshot: path=/tmp/raft904014667/snapshots/2-2-1750914776762.tmp
2025-06-26T05:12:56.762Z [INFO]  server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82: starting restore from snapshot: id=2-2-1750914776762 last-index=2 last-term=2 size-in-bytes=1
2025-06-26T05:12:56.762Z [INFO]  server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82: snapshot restore progress: id=2-2-1750914776762 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-06-26T05:12:56.762Z [INFO]  server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82: restored from snapshot: id=2-2-1750914776762 last-index=2 last-term=2 size-in-bytes=1
2025-06-26T05:12:56.762Z [INFO]  server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3912c67-83c5-ac1e-b30e-ca9a662beb19 Address:a3912c67-83c5-ac1e-b30e-ca9a662beb19} {Suffrage:Voter ID:server-e289064c-cae2-0623-0617-1cc79076933a Address:e289064c-cae2-0623-0617-1cc79076933a} {Suffrage:Voter ID:server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82 Address:76be5bdf-1aab-d840-2219-ea5f3cb0bb82}]"
2025-06-26T05:12:56.762Z [INFO]  server-a3912c67-83c5-ac1e-b30e-ca9a662beb19: entering follower state: follower="Node at a3912c67-83c5-ac1e-b30e-ca9a662beb19 [Follower]" leader-address= leader-id=
2025-06-26T05:12:56.762Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: entering follower state: follower="Node at e289064c-cae2-0623-0617-1cc79076933a [Follower]" leader-address= leader-id=
2025-06-26T05:12:56.762Z [INFO]  server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82: entering follower state: follower="Node at 76be5bdf-1aab-d840-2219-ea5f3cb0bb82 [Follower]" leader-address= leader-id=
2025-06-26T05:12:56.841Z [WARN]  server-e289064c-cae2-0623-0617-1cc79076933a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:56.841Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: entering candidate state: node="Node at e289064c-cae2-0623-0617-1cc79076933a [Candidate]" term=3
2025-06-26T05:12:56.841Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: election won: term=3 tally=2
2025-06-26T05:12:56.841Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: entering leader state: leader="Node at e289064c-cae2-0623-0617-1cc79076933a [Leader]"
2025-06-26T05:12:56.841Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: added peer, starting replication: peer=server-a3912c67-83c5-ac1e-b30e-ca9a662beb19
2025-06-26T05:12:56.841Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: added peer, starting replication: peer=server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82
2025-06-26T05:12:56.841Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: pipelining replication: peer="{Voter server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82 76be5bdf-1aab-d840-2219-ea5f3cb0bb82}"
2025-06-26T05:12:56.841Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: pipelining replication: peer="{Voter server-a3912c67-83c5-ac1e-b30e-ca9a662beb19 a3912c67-83c5-ac1e-b30e-ca9a662beb19}"
2025-06-26T05:12:57.081Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:57.189Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:57.081791243 +0000 UTC m=+5.230901521 (1 nodes), 17.6µs from start of poll, 519.26516ms from cluster start. Timeout at 2025-06-26 05:12:57.189684803 +0000 UTC m=+5.338795081, 107.89356ms after stability
2025-06-26T05:12:57.189Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: aborting pipeline replication: peer="{Voter server-76be5bdf-1aab-d840-2219-ea5f3cb0bb82 76be5bdf-1aab-d840-2219-ea5f3cb0bb82}"
2025-06-26T05:12:57.189Z [INFO]  server-e289064c-cae2-0623-0617-1cc79076933a: aborting pipeline replication: peer="{Voter server-a3912c67-83c5-ac1e-b30e-ca9a662beb19 a3912c67-83c5-ac1e-b30e-ca9a662beb19}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2025-06-26T05:12:57.190Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17f35b0d-31b3-f27a-48e9-18a2f61a0432 Address:17f35b0d-31b3-f27a-48e9-18a2f61a0432} {Suffrage:Voter ID:server-4ba95847-1448-fcf1-a94e-c8571a22b92b Address:4ba95847-1448-fcf1-a94e-c8571a22b92b} {Suffrage:Voter ID:server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc Address:9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc}]"
2025-06-26T05:12:57.190Z [INFO]  server-4ba95847-1448-fcf1-a94e-c8571a22b92b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17f35b0d-31b3-f27a-48e9-18a2f61a0432 Address:17f35b0d-31b3-f27a-48e9-18a2f61a0432} {Suffrage:Voter ID:server-4ba95847-1448-fcf1-a94e-c8571a22b92b Address:4ba95847-1448-fcf1-a94e-c8571a22b92b} {Suffrage:Voter ID:server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc Address:9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc}]"
2025-06-26T05:12:57.190Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17f35b0d-31b3-f27a-48e9-18a2f61a0432 Address:17f35b0d-31b3-f27a-48e9-18a2f61a0432} {Suffrage:Voter ID:server-4ba95847-1448-fcf1-a94e-c8571a22b92b Address:4ba95847-1448-fcf1-a94e-c8571a22b92b} {Suffrage:Voter ID:server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc Address:9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc}]"
2025-06-26T05:12:57.190Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:57.190Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: entering follower state: follower="Node at 17f35b0d-31b3-f27a-48e9-18a2f61a0432 [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.190Z [INFO]  server-4ba95847-1448-fcf1-a94e-c8571a22b92b: entering follower state: follower="Node at 4ba95847-1448-fcf1-a94e-c8571a22b92b [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.190Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: entering follower state: follower="Node at 9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.249Z [WARN]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:57.249Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: entering candidate state: node="Node at 17f35b0d-31b3-f27a-48e9-18a2f61a0432 [Candidate]" term=2
2025-06-26T05:12:57.249Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: election won: term=2 tally=2
2025-06-26T05:12:57.249Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: entering leader state: leader="Node at 17f35b0d-31b3-f27a-48e9-18a2f61a0432 [Leader]"
2025-06-26T05:12:57.249Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: added peer, starting replication: peer=server-4ba95847-1448-fcf1-a94e-c8571a22b92b
2025-06-26T05:12:57.249Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: added peer, starting replication: peer=server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc
2025-06-26T05:12:57.249Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: pipelining replication: peer="{Voter server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc 9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc}"
2025-06-26T05:12:57.249Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: pipelining replication: peer="{Voter server-4ba95847-1448-fcf1-a94e-c8571a22b92b 4ba95847-1448-fcf1-a94e-c8571a22b92b}"
2025-06-26T05:12:57.357Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:57.249971963 +0000 UTC m=+5.399082241 (1 nodes), 59.54328ms from start of poll, 59.7366ms from cluster start. Timeout at 2025-06-26 05:12:57.357605163 +0000 UTC m=+5.506715441, 107.6332ms after stability
2025-06-26T05:12:57.358Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: aborting pipeline replication: peer="{Voter server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc 9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc}"
2025-06-26T05:12:57.358Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: aborting pipeline replication: peer="{Voter server-4ba95847-1448-fcf1-a94e-c8571a22b92b 4ba95847-1448-fcf1-a94e-c8571a22b92b}"
2025-06-26T05:12:57.358Z [INFO]  creating new snapshot: path=/tmp/raft2335624068/snapshots/2-6-1750914777358.tmp
2025-06-26T05:12:57.358Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: starting restore from snapshot: id=2-6-1750914777358 last-index=6 last-term=2 size-in-bytes=25
2025-06-26T05:12:57.358Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: snapshot restore progress: id=2-6-1750914777358 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-06-26T05:12:57.358Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: restored from snapshot: id=2-6-1750914777358 last-index=6 last-term=2 size-in-bytes=25
2025-06-26T05:12:57.358Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17f35b0d-31b3-f27a-48e9-18a2f61a0432 Address:17f35b0d-31b3-f27a-48e9-18a2f61a0432} {Suffrage:Voter ID:server-4ba95847-1448-fcf1-a94e-c8571a22b92b Address:4ba95847-1448-fcf1-a94e-c8571a22b92b} {Suffrage:Voter ID:server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc Address:9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc}]"
2025-06-26T05:12:57.358Z [INFO]  creating new snapshot: path=/tmp/raft3958565317/snapshots/2-6-1750914777358.tmp
2025-06-26T05:12:57.358Z [INFO]  server-4ba95847-1448-fcf1-a94e-c8571a22b92b: starting restore from snapshot: id=2-6-1750914777358 last-index=6 last-term=2 size-in-bytes=25
2025-06-26T05:12:57.358Z [INFO]  server-4ba95847-1448-fcf1-a94e-c8571a22b92b: snapshot restore progress: id=2-6-1750914777358 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-06-26T05:12:57.358Z [INFO]  server-4ba95847-1448-fcf1-a94e-c8571a22b92b: restored from snapshot: id=2-6-1750914777358 last-index=6 last-term=2 size-in-bytes=25
2025-06-26T05:12:57.358Z [INFO]  server-4ba95847-1448-fcf1-a94e-c8571a22b92b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17f35b0d-31b3-f27a-48e9-18a2f61a0432 Address:17f35b0d-31b3-f27a-48e9-18a2f61a0432} {Suffrage:Voter ID:server-4ba95847-1448-fcf1-a94e-c8571a22b92b Address:4ba95847-1448-fcf1-a94e-c8571a22b92b} {Suffrage:Voter ID:server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc Address:9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc}]"
2025-06-26T05:12:57.358Z [INFO]  creating new snapshot: path=/tmp/raft3935565814/snapshots/2-6-1750914777358.tmp
2025-06-26T05:12:57.359Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: starting restore from snapshot: id=2-6-1750914777358 last-index=6 last-term=2 size-in-bytes=25
2025-06-26T05:12:57.359Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: snapshot restore progress: id=2-6-1750914777358 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-06-26T05:12:57.359Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: restored from snapshot: id=2-6-1750914777358 last-index=6 last-term=2 size-in-bytes=25
2025-06-26T05:12:57.359Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17f35b0d-31b3-f27a-48e9-18a2f61a0432 Address:17f35b0d-31b3-f27a-48e9-18a2f61a0432} {Suffrage:Voter ID:server-4ba95847-1448-fcf1-a94e-c8571a22b92b Address:4ba95847-1448-fcf1-a94e-c8571a22b92b} {Suffrage:Voter ID:server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc Address:9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc}]"
2025-06-26T05:12:57.359Z [INFO]  server-17f35b0d-31b3-f27a-48e9-18a2f61a0432: entering follower state: follower="Node at 17f35b0d-31b3-f27a-48e9-18a2f61a0432 [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.359Z [INFO]  server-4ba95847-1448-fcf1-a94e-c8571a22b92b: entering follower state: follower="Node at 4ba95847-1448-fcf1-a94e-c8571a22b92b [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.359Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: entering follower state: follower="Node at 9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.417Z [WARN]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:57.417Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: entering candidate state: node="Node at 9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc [Candidate]" term=3
2025-06-26T05:12:57.417Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: election won: term=3 tally=2
2025-06-26T05:12:57.417Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: entering leader state: leader="Node at 9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc [Leader]"
2025-06-26T05:12:57.417Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: added peer, starting replication: peer=server-17f35b0d-31b3-f27a-48e9-18a2f61a0432
2025-06-26T05:12:57.417Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: added peer, starting replication: peer=server-4ba95847-1448-fcf1-a94e-c8571a22b92b
2025-06-26T05:12:57.417Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: pipelining replication: peer="{Voter server-4ba95847-1448-fcf1-a94e-c8571a22b92b 4ba95847-1448-fcf1-a94e-c8571a22b92b}"
2025-06-26T05:12:57.417Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: pipelining replication: peer="{Voter server-17f35b0d-31b3-f27a-48e9-18a2f61a0432 17f35b0d-31b3-f27a-48e9-18a2f61a0432}"
2025-06-26T05:12:57.677Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:57.785Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:57.677732563 +0000 UTC m=+5.826842841 (1 nodes), 20.88µs from start of poll, 487.4972ms from cluster start. Timeout at 2025-06-26 05:12:57.785600923 +0000 UTC m=+5.934711201, 107.86836ms after stability
2025-06-26T05:12:57.785Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: aborting pipeline replication: peer="{Voter server-4ba95847-1448-fcf1-a94e-c8571a22b92b 4ba95847-1448-fcf1-a94e-c8571a22b92b}"
2025-06-26T05:12:57.785Z [INFO]  server-9fedb38f-bc13-4ea4-6a08-7fd3bdc65fbc: aborting pipeline replication: peer="{Voter server-17f35b0d-31b3-f27a-48e9-18a2f61a0432 17f35b0d-31b3-f27a-48e9-18a2f61a0432}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2025-06-26T05:12:57.790Z [INFO]  server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331 Address:5c9e5cdf-8a9a-b008-3a2b-38af6e301331} {Suffrage:Voter ID:server-72961622-08ad-ceab-22e0-1093e3aea20d Address:72961622-08ad-ceab-22e0-1093e3aea20d} {Suffrage:Voter ID:server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 Address:32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50}]"
2025-06-26T05:12:57.790Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331 Address:5c9e5cdf-8a9a-b008-3a2b-38af6e301331} {Suffrage:Voter ID:server-72961622-08ad-ceab-22e0-1093e3aea20d Address:72961622-08ad-ceab-22e0-1093e3aea20d} {Suffrage:Voter ID:server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 Address:32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50}]"
2025-06-26T05:12:57.790Z [INFO]  server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331 Address:5c9e5cdf-8a9a-b008-3a2b-38af6e301331} {Suffrage:Voter ID:server-72961622-08ad-ceab-22e0-1093e3aea20d Address:72961622-08ad-ceab-22e0-1093e3aea20d} {Suffrage:Voter ID:server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 Address:32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50}]"
2025-06-26T05:12:57.790Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:57.790Z [INFO]  server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331: entering follower state: follower="Node at 5c9e5cdf-8a9a-b008-3a2b-38af6e301331 [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.790Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: entering follower state: follower="Node at 72961622-08ad-ceab-22e0-1093e3aea20d [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.790Z [INFO]  server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50: entering follower state: follower="Node at 32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.845Z [WARN]  server-72961622-08ad-ceab-22e0-1093e3aea20d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:57.845Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: entering candidate state: node="Node at 72961622-08ad-ceab-22e0-1093e3aea20d [Candidate]" term=2
2025-06-26T05:12:57.845Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: election won: term=2 tally=2
2025-06-26T05:12:57.845Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: entering leader state: leader="Node at 72961622-08ad-ceab-22e0-1093e3aea20d [Leader]"
2025-06-26T05:12:57.845Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: added peer, starting replication: peer=server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331
2025-06-26T05:12:57.845Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: added peer, starting replication: peer=server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50
2025-06-26T05:12:57.845Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: pipelining replication: peer="{Voter server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50}"
2025-06-26T05:12:57.846Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: pipelining replication: peer="{Voter server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331 5c9e5cdf-8a9a-b008-3a2b-38af6e301331}"
2025-06-26T05:12:57.953Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:57.846039483 +0000 UTC m=+5.995149721 (1 nodes), 55.80928ms from start of poll, 56.10208ms from cluster start. Timeout at 2025-06-26 05:12:57.953612563 +0000 UTC m=+6.102722921, 107.5732ms after stability
2025-06-26T05:12:57.955Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: aborting pipeline replication: peer="{Voter server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50}"
2025-06-26T05:12:57.955Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: aborting pipeline replication: peer="{Voter server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331 5c9e5cdf-8a9a-b008-3a2b-38af6e301331}"
2025-06-26T05:12:57.955Z [INFO]  creating new snapshot: path=/tmp/raft2729070743/snapshots/2-27-1750914777955.tmp
2025-06-26T05:12:57.956Z [INFO]  server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331: starting restore from snapshot: id=2-27-1750914777955 last-index=27 last-term=2 size-in-bytes=168
2025-06-26T05:12:57.956Z [INFO]  server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331: snapshot restore progress: id=2-27-1750914777955 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-06-26T05:12:57.956Z [INFO]  server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331: restored from snapshot: id=2-27-1750914777955 last-index=27 last-term=2 size-in-bytes=168
2025-06-26T05:12:57.956Z [INFO]  server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331 Address:5c9e5cdf-8a9a-b008-3a2b-38af6e301331} {Suffrage:Voter ID:server-72961622-08ad-ceab-22e0-1093e3aea20d Address:72961622-08ad-ceab-22e0-1093e3aea20d} {Suffrage:Voter ID:server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 Address:32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50}]"
2025-06-26T05:12:57.956Z [INFO]  creating new snapshot: path=/tmp/raft3118716840/snapshots/2-27-1750914777956.tmp
2025-06-26T05:12:57.956Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: starting restore from snapshot: id=2-27-1750914777956 last-index=27 last-term=2 size-in-bytes=168
2025-06-26T05:12:57.956Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: snapshot restore progress: id=2-27-1750914777956 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-06-26T05:12:57.956Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: restored from snapshot: id=2-27-1750914777956 last-index=27 last-term=2 size-in-bytes=168
2025-06-26T05:12:57.956Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331 Address:5c9e5cdf-8a9a-b008-3a2b-38af6e301331} {Suffrage:Voter ID:server-72961622-08ad-ceab-22e0-1093e3aea20d Address:72961622-08ad-ceab-22e0-1093e3aea20d} {Suffrage:Voter ID:server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 Address:32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50}]"
2025-06-26T05:12:57.956Z [INFO]  creating new snapshot: path=/tmp/raft3040450247/snapshots/2-27-1750914777956.tmp
2025-06-26T05:12:57.956Z [INFO]  server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50: starting restore from snapshot: id=2-27-1750914777956 last-index=27 last-term=2 size-in-bytes=168
2025-06-26T05:12:57.956Z [INFO]  server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50: snapshot restore progress: id=2-27-1750914777956 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-06-26T05:12:57.956Z [INFO]  server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50: restored from snapshot: id=2-27-1750914777956 last-index=27 last-term=2 size-in-bytes=168
2025-06-26T05:12:57.956Z [INFO]  server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331 Address:5c9e5cdf-8a9a-b008-3a2b-38af6e301331} {Suffrage:Voter ID:server-72961622-08ad-ceab-22e0-1093e3aea20d Address:72961622-08ad-ceab-22e0-1093e3aea20d} {Suffrage:Voter ID:server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 Address:32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50}]"
2025-06-26T05:12:57.956Z [INFO]  server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331: entering follower state: follower="Node at 5c9e5cdf-8a9a-b008-3a2b-38af6e301331 [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.956Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: entering follower state: follower="Node at 72961622-08ad-ceab-22e0-1093e3aea20d [Follower]" leader-address= leader-id=
2025-06-26T05:12:57.956Z [INFO]  server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50: entering follower state: follower="Node at 32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 [Follower]" leader-address= leader-id=
2025-06-26T05:12:58.025Z [WARN]  server-72961622-08ad-ceab-22e0-1093e3aea20d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:58.025Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: entering candidate state: node="Node at 72961622-08ad-ceab-22e0-1093e3aea20d [Candidate]" term=3
2025-06-26T05:12:58.025Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: election won: term=3 tally=2
2025-06-26T05:12:58.025Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: entering leader state: leader="Node at 72961622-08ad-ceab-22e0-1093e3aea20d [Leader]"
2025-06-26T05:12:58.025Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: added peer, starting replication: peer=server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331
2025-06-26T05:12:58.025Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: added peer, starting replication: peer=server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50
2025-06-26T05:12:58.025Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: pipelining replication: peer="{Voter server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50}"
2025-06-26T05:12:58.025Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: pipelining replication: peer="{Voter server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331 5c9e5cdf-8a9a-b008-3a2b-38af6e301331}"
2025-06-26T05:12:58.273Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:58.381Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:58.273686963 +0000 UTC m=+6.422797241 (1 nodes), 21.4µs from start of poll, 483.7496ms from cluster start. Timeout at 2025-06-26 05:12:58.381702683 +0000 UTC m=+6.530812961, 108.01572ms after stability
2025-06-26T05:12:58.381Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: aborting pipeline replication: peer="{Voter server-32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50 32a40b4f-ad4d-0b74-6409-f4dfeb4a9c50}"
2025-06-26T05:12:58.381Z [INFO]  server-72961622-08ad-ceab-22e0-1093e3aea20d: aborting pipeline replication: peer="{Voter server-5c9e5cdf-8a9a-b008-3a2b-38af6e301331 5c9e5cdf-8a9a-b008-3a2b-38af6e301331}"
--- PASS: TestRaft_RecoverCluster (1.82s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.63s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.60s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.60s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2025-06-26T05:12:58.382Z [INFO]  server-60dcf5a5-54de-f717-8d48-e3362078e5c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60dcf5a5-54de-f717-8d48-e3362078e5c9 Address:60dcf5a5-54de-f717-8d48-e3362078e5c9}]"
2025-06-26T05:12:58.382Z [INFO]  server-60dcf5a5-54de-f717-8d48-e3362078e5c9: entering follower state: follower="Node at 60dcf5a5-54de-f717-8d48-e3362078e5c9 [Follower]" leader-address= leader-id=
2025-06-26T05:12:58.437Z [WARN]  server-60dcf5a5-54de-f717-8d48-e3362078e5c9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:58.437Z [INFO]  server-60dcf5a5-54de-f717-8d48-e3362078e5c9: entering candidate state: node="Node at 60dcf5a5-54de-f717-8d48-e3362078e5c9 [Candidate]" term=2
2025-06-26T05:12:58.437Z [INFO]  server-60dcf5a5-54de-f717-8d48-e3362078e5c9: election won: term=2 tally=1
2025-06-26T05:12:58.437Z [INFO]  server-60dcf5a5-54de-f717-8d48-e3362078e5c9: entering leader state: leader="Node at 60dcf5a5-54de-f717-8d48-e3362078e5c9 [Leader]"
--- PASS: TestRaft_SingleNode (0.06s)
=== RUN   TestRaft_TripleNode
2025-06-26T05:12:58.438Z [INFO]  server-20d4c87d-f50e-429c-3a50-5f49eb2e2dfc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20d4c87d-f50e-429c-3a50-5f49eb2e2dfc Address:20d4c87d-f50e-429c-3a50-5f49eb2e2dfc} {Suffrage:Voter ID:server-666c8df5-5c14-50aa-a4fb-651d0b9cfb8c Address:666c8df5-5c14-50aa-a4fb-651d0b9cfb8c} {Suffrage:Voter ID:server-06f10410-ef36-51cd-e225-07b628a7e69b Address:06f10410-ef36-51cd-e225-07b628a7e69b}]"
2025-06-26T05:12:58.438Z [INFO]  server-666c8df5-5c14-50aa-a4fb-651d0b9cfb8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20d4c87d-f50e-429c-3a50-5f49eb2e2dfc Address:20d4c87d-f50e-429c-3a50-5f49eb2e2dfc} {Suffrage:Voter ID:server-666c8df5-5c14-50aa-a4fb-651d0b9cfb8c Address:666c8df5-5c14-50aa-a4fb-651d0b9cfb8c} {Suffrage:Voter ID:server-06f10410-ef36-51cd-e225-07b628a7e69b Address:06f10410-ef36-51cd-e225-07b628a7e69b}]"
2025-06-26T05:12:58.438Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20d4c87d-f50e-429c-3a50-5f49eb2e2dfc Address:20d4c87d-f50e-429c-3a50-5f49eb2e2dfc} {Suffrage:Voter ID:server-666c8df5-5c14-50aa-a4fb-651d0b9cfb8c Address:666c8df5-5c14-50aa-a4fb-651d0b9cfb8c} {Suffrage:Voter ID:server-06f10410-ef36-51cd-e225-07b628a7e69b Address:06f10410-ef36-51cd-e225-07b628a7e69b}]"
2025-06-26T05:12:58.438Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:12:58.438Z [INFO]  server-20d4c87d-f50e-429c-3a50-5f49eb2e2dfc: entering follower state: follower="Node at 20d4c87d-f50e-429c-3a50-5f49eb2e2dfc [Follower]" leader-address= leader-id=
2025-06-26T05:12:58.438Z [INFO]  server-666c8df5-5c14-50aa-a4fb-651d0b9cfb8c: entering follower state: follower="Node at 666c8df5-5c14-50aa-a4fb-651d0b9cfb8c [Follower]" leader-address= leader-id=
2025-06-26T05:12:58.438Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: entering follower state: follower="Node at 06f10410-ef36-51cd-e225-07b628a7e69b [Follower]" leader-address= leader-id=
2025-06-26T05:12:58.489Z [WARN]  server-06f10410-ef36-51cd-e225-07b628a7e69b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:58.489Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: entering candidate state: node="Node at 06f10410-ef36-51cd-e225-07b628a7e69b [Candidate]" term=2
2025-06-26T05:12:58.489Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: election won: term=2 tally=2
2025-06-26T05:12:58.489Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: entering leader state: leader="Node at 06f10410-ef36-51cd-e225-07b628a7e69b [Leader]"
2025-06-26T05:12:58.489Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: added peer, starting replication: peer=server-20d4c87d-f50e-429c-3a50-5f49eb2e2dfc
2025-06-26T05:12:58.489Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: added peer, starting replication: peer=server-666c8df5-5c14-50aa-a4fb-651d0b9cfb8c
2025-06-26T05:12:58.489Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: pipelining replication: peer="{Voter server-666c8df5-5c14-50aa-a4fb-651d0b9cfb8c 666c8df5-5c14-50aa-a4fb-651d0b9cfb8c}"
2025-06-26T05:12:58.489Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: pipelining replication: peer="{Voter server-20d4c87d-f50e-429c-3a50-5f49eb2e2dfc 20d4c87d-f50e-429c-3a50-5f49eb2e2dfc}"
2025-06-26T05:12:58.601Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:12:58.489966803 +0000 UTC m=+6.639077081 (2 nodes), 51.7314ms from start of poll, 51.9168ms from cluster start. Timeout at 2025-06-26 05:12:58.601599043 +0000 UTC m=+6.750709321, 111.63224ms after stability
2025-06-26T05:12:58.601Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:58.713Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:58.601720123 +0000 UTC m=+6.750830401 (1 nodes), 1.56µs from start of poll, 163.67012ms from cluster start. Timeout at 2025-06-26 05:12:58.713627003 +0000 UTC m=+6.862737281, 111.90688ms after stability
2025-06-26T05:12:58.729Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: aborting pipeline replication: peer="{Voter server-666c8df5-5c14-50aa-a4fb-651d0b9cfb8c 666c8df5-5c14-50aa-a4fb-651d0b9cfb8c}"
2025-06-26T05:12:58.729Z [INFO]  server-06f10410-ef36-51cd-e225-07b628a7e69b: aborting pipeline replication: peer="{Voter server-20d4c87d-f50e-429c-3a50-5f49eb2e2dfc 20d4c87d-f50e-429c-3a50-5f49eb2e2dfc}"
2025-06-26T05:12:59.233Z [ERROR] server-06f10410-ef36-51cd-e225-07b628a7e69b: failed to heartbeat to: peer=666c8df5-5c14-50aa-a4fb-651d0b9cfb8c backoff time=10ms error="command timed out"
--- PASS: TestRaft_TripleNode (0.80s)
=== RUN   TestRaft_LeaderFail
2025-06-26T05:12:59.234Z [INFO]  server-2c7fd39a-2e10-b227-9700-82e35428c82b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7fd39a-2e10-b227-9700-82e35428c82b Address:2c7fd39a-2e10-b227-9700-82e35428c82b} {Suffrage:Voter ID:server-49d94997-6610-92f1-c1c3-a0e48f2e4492 Address:49d94997-6610-92f1-c1c3-a0e48f2e4492} {Suffrage:Voter ID:server-42f4a127-9a79-4785-8778-8e9e20e48587 Address:42f4a127-9a79-4785-8778-8e9e20e48587}]"
2025-06-26T05:12:59.234Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7fd39a-2e10-b227-9700-82e35428c82b Address:2c7fd39a-2e10-b227-9700-82e35428c82b} {Suffrage:Voter ID:server-49d94997-6610-92f1-c1c3-a0e48f2e4492 Address:49d94997-6610-92f1-c1c3-a0e48f2e4492} {Suffrage:Voter ID:server-42f4a127-9a79-4785-8778-8e9e20e48587 Address:42f4a127-9a79-4785-8778-8e9e20e48587}]"
2025-06-26T05:12:59.234Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7fd39a-2e10-b227-9700-82e35428c82b Address:2c7fd39a-2e10-b227-9700-82e35428c82b} {Suffrage:Voter ID:server-49d94997-6610-92f1-c1c3-a0e48f2e4492 Address:49d94997-6610-92f1-c1c3-a0e48f2e4492} {Suffrage:Voter ID:server-42f4a127-9a79-4785-8778-8e9e20e48587 Address:42f4a127-9a79-4785-8778-8e9e20e48587}]"
2025-06-26T05:12:59.234Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:12:59.234Z [INFO]  server-2c7fd39a-2e10-b227-9700-82e35428c82b: entering follower state: follower="Node at 2c7fd39a-2e10-b227-9700-82e35428c82b [Follower]" leader-address= leader-id=
2025-06-26T05:12:59.234Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: entering follower state: follower="Node at 49d94997-6610-92f1-c1c3-a0e48f2e4492 [Follower]" leader-address= leader-id=
2025-06-26T05:12:59.234Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: entering follower state: follower="Node at 42f4a127-9a79-4785-8778-8e9e20e48587 [Follower]" leader-address= leader-id=
2025-06-26T05:12:59.289Z [WARN]  server-42f4a127-9a79-4785-8778-8e9e20e48587: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:59.289Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: entering candidate state: node="Node at 42f4a127-9a79-4785-8778-8e9e20e48587 [Candidate]" term=2
2025-06-26T05:12:59.289Z [WARN]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:59.289Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: entering candidate state: node="Node at 49d94997-6610-92f1-c1c3-a0e48f2e4492 [Candidate]" term=2
2025-06-26T05:12:59.289Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: duplicate requestVote for same term: term=2
2025-06-26T05:12:59.289Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: election won: term=2 tally=2
2025-06-26T05:12:59.289Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: entering leader state: leader="Node at 42f4a127-9a79-4785-8778-8e9e20e48587 [Leader]"
2025-06-26T05:12:59.289Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: added peer, starting replication: peer=server-2c7fd39a-2e10-b227-9700-82e35428c82b
2025-06-26T05:12:59.289Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: added peer, starting replication: peer=server-49d94997-6610-92f1-c1c3-a0e48f2e4492
2025-06-26T05:12:59.289Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: entering follower state: follower="Node at 49d94997-6610-92f1-c1c3-a0e48f2e4492 [Follower]" leader-address=42f4a127-9a79-4785-8778-8e9e20e48587 leader-id=server-42f4a127-9a79-4785-8778-8e9e20e48587
2025-06-26T05:12:59.289Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: pipelining replication: peer="{Voter server-49d94997-6610-92f1-c1c3-a0e48f2e4492 49d94997-6610-92f1-c1c3-a0e48f2e4492}"
2025-06-26T05:12:59.289Z [INFO]  server-2c7fd39a-2e10-b227-9700-82e35428c82b: duplicate requestVote for same term: term=2
2025-06-26T05:12:59.289Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: pipelining replication: peer="{Voter server-2c7fd39a-2e10-b227-9700-82e35428c82b 2c7fd39a-2e10-b227-9700-82e35428c82b}"
2025-06-26T05:12:59.289Z [WARN]  server-42f4a127-9a79-4785-8778-8e9e20e48587: rejecting vote request since we have a leader: from=49d94997-6610-92f1-c1c3-a0e48f2e4492 leader=42f4a127-9a79-4785-8778-8e9e20e48587 leader-id=server-42f4a127-9a79-4785-8778-8e9e20e48587
2025-06-26T05:12:59.397Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:12:59.289956483 +0000 UTC m=+7.439066761 (2 nodes), 55.77368ms from start of poll, 55.96524ms from cluster start. Timeout at 2025-06-26 05:12:59.397598323 +0000 UTC m=+7.546708601, 107.64184ms after stability
2025-06-26T05:12:59.397Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:59.505Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:59.397700523 +0000 UTC m=+7.546810801 (1 nodes), 1.72µs from start of poll, 163.70928ms from cluster start. Timeout at 2025-06-26 05:12:59.505581723 +0000 UTC m=+7.654692001, 107.8812ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at 42f4a127-9a79-4785-8778-8e9e20e48587 [Leader]
2025-06-26T05:12:59.533Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to heartbeat to: peer=2c7fd39a-2e10-b227-9700-82e35428c82b backoff time=10ms error="failed to connect to peer: 2c7fd39a-2e10-b227-9700-82e35428c82b"
2025-06-26T05:12:59.533Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:59.533Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to heartbeat to: peer=49d94997-6610-92f1-c1c3-a0e48f2e4492 backoff time=10ms error="failed to connect to peer: 49d94997-6610-92f1-c1c3-a0e48f2e4492"
2025-06-26T05:12:59.537Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to pipeline appendEntries: peer="{Voter server-2c7fd39a-2e10-b227-9700-82e35428c82b 2c7fd39a-2e10-b227-9700-82e35428c82b}" error="append pipeline closed"
2025-06-26T05:12:59.537Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: aborting pipeline replication: peer="{Voter server-2c7fd39a-2e10-b227-9700-82e35428c82b 2c7fd39a-2e10-b227-9700-82e35428c82b}"
2025-06-26T05:12:59.541Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to pipeline appendEntries: peer="{Voter server-49d94997-6610-92f1-c1c3-a0e48f2e4492 49d94997-6610-92f1-c1c3-a0e48f2e4492}" error="append pipeline closed"
2025-06-26T05:12:59.541Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: aborting pipeline replication: peer="{Voter server-49d94997-6610-92f1-c1c3-a0e48f2e4492 49d94997-6610-92f1-c1c3-a0e48f2e4492}"
2025-06-26T05:12:59.545Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to appendEntries to: peer="{Voter server-2c7fd39a-2e10-b227-9700-82e35428c82b 2c7fd39a-2e10-b227-9700-82e35428c82b}" error="failed to connect to peer: 2c7fd39a-2e10-b227-9700-82e35428c82b"
2025-06-26T05:12:59.553Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to heartbeat to: peer=49d94997-6610-92f1-c1c3-a0e48f2e4492 backoff time=10ms error="failed to connect to peer: 49d94997-6610-92f1-c1c3-a0e48f2e4492"
2025-06-26T05:12:59.553Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to appendEntries to: peer="{Voter server-49d94997-6610-92f1-c1c3-a0e48f2e4492 49d94997-6610-92f1-c1c3-a0e48f2e4492}" error="failed to connect to peer: 49d94997-6610-92f1-c1c3-a0e48f2e4492"
2025-06-26T05:12:59.557Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to heartbeat to: peer=2c7fd39a-2e10-b227-9700-82e35428c82b backoff time=10ms error="failed to connect to peer: 2c7fd39a-2e10-b227-9700-82e35428c82b"
2025-06-26T05:12:59.565Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to appendEntries to: peer="{Voter server-2c7fd39a-2e10-b227-9700-82e35428c82b 2c7fd39a-2e10-b227-9700-82e35428c82b}" error="failed to connect to peer: 2c7fd39a-2e10-b227-9700-82e35428c82b"
2025-06-26T05:12:59.573Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to heartbeat to: peer=49d94997-6610-92f1-c1c3-a0e48f2e4492 backoff time=10ms error="failed to connect to peer: 49d94997-6610-92f1-c1c3-a0e48f2e4492"
2025-06-26T05:12:59.577Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to appendEntries to: peer="{Voter server-49d94997-6610-92f1-c1c3-a0e48f2e4492 49d94997-6610-92f1-c1c3-a0e48f2e4492}" error="failed to connect to peer: 49d94997-6610-92f1-c1c3-a0e48f2e4492"
2025-06-26T05:12:59.581Z [WARN]  server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to contact: server-id=server-2c7fd39a-2e10-b227-9700-82e35428c82b time=51.95148ms
2025-06-26T05:12:59.581Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to heartbeat to: peer=2c7fd39a-2e10-b227-9700-82e35428c82b backoff time=10ms error="failed to connect to peer: 2c7fd39a-2e10-b227-9700-82e35428c82b"
2025-06-26T05:12:59.581Z [WARN]  server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to contact: server-id=server-49d94997-6610-92f1-c1c3-a0e48f2e4492 time=51.96936ms
2025-06-26T05:12:59.581Z [WARN]  server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to contact quorum of nodes, stepping down
2025-06-26T05:12:59.581Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: entering follower state: follower="Node at 42f4a127-9a79-4785-8778-8e9e20e48587 [Follower]" leader-address= leader-id=
2025-06-26T05:12:59.589Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to appendEntries to: peer="{Voter server-2c7fd39a-2e10-b227-9700-82e35428c82b 2c7fd39a-2e10-b227-9700-82e35428c82b}" error="failed to connect to peer: 2c7fd39a-2e10-b227-9700-82e35428c82b"
2025-06-26T05:12:59.593Z [WARN]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: heartbeat timeout reached, starting election: last-leader-addr=42f4a127-9a79-4785-8778-8e9e20e48587 last-leader-id=server-42f4a127-9a79-4785-8778-8e9e20e48587
2025-06-26T05:12:59.593Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: entering candidate state: node="Node at 49d94997-6610-92f1-c1c3-a0e48f2e4492 [Candidate]" term=3
2025-06-26T05:12:59.593Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to make requestVote RPC: target="{Voter server-42f4a127-9a79-4785-8778-8e9e20e48587 42f4a127-9a79-4785-8778-8e9e20e48587}" error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587" term=3
2025-06-26T05:12:59.593Z [WARN]  server-2c7fd39a-2e10-b227-9700-82e35428c82b: rejecting vote request since we have a leader: from=49d94997-6610-92f1-c1c3-a0e48f2e4492 leader=42f4a127-9a79-4785-8778-8e9e20e48587 leader-id=server-42f4a127-9a79-4785-8778-8e9e20e48587
2025-06-26T05:12:59.601Z [WARN]  server-2c7fd39a-2e10-b227-9700-82e35428c82b: heartbeat timeout reached, starting election: last-leader-addr=42f4a127-9a79-4785-8778-8e9e20e48587 last-leader-id=server-42f4a127-9a79-4785-8778-8e9e20e48587
2025-06-26T05:12:59.601Z [INFO]  server-2c7fd39a-2e10-b227-9700-82e35428c82b: entering candidate state: node="Node at 2c7fd39a-2e10-b227-9700-82e35428c82b [Candidate]" term=3
2025-06-26T05:12:59.601Z [ERROR] server-2c7fd39a-2e10-b227-9700-82e35428c82b: failed to make requestVote RPC: target="{Voter server-42f4a127-9a79-4785-8778-8e9e20e48587 42f4a127-9a79-4785-8778-8e9e20e48587}" error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587" term=3
2025-06-26T05:12:59.601Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: duplicate requestVote for same term: term=3
2025-06-26T05:12:59.677Z [WARN]  server-42f4a127-9a79-4785-8778-8e9e20e48587: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:59.677Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: entering candidate state: node="Node at 42f4a127-9a79-4785-8778-8e9e20e48587 [Candidate]" term=3
2025-06-26T05:12:59.677Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to make requestVote RPC: target="{Voter server-49d94997-6610-92f1-c1c3-a0e48f2e4492 49d94997-6610-92f1-c1c3-a0e48f2e4492}" error="failed to connect to peer: 49d94997-6610-92f1-c1c3-a0e48f2e4492" term=3
2025-06-26T05:12:59.677Z [WARN]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: Election timeout reached, restarting election
2025-06-26T05:12:59.677Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: entering candidate state: node="Node at 49d94997-6610-92f1-c1c3-a0e48f2e4492 [Candidate]" term=4
2025-06-26T05:12:59.677Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to make requestVote RPC: target="{Voter server-42f4a127-9a79-4785-8778-8e9e20e48587 42f4a127-9a79-4785-8778-8e9e20e48587}" error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587" term=4
2025-06-26T05:12:59.677Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to make requestVote RPC: target="{Voter server-2c7fd39a-2e10-b227-9700-82e35428c82b 2c7fd39a-2e10-b227-9700-82e35428c82b}" error="failed to connect to peer: 2c7fd39a-2e10-b227-9700-82e35428c82b" term=3
2025-06-26T05:12:59.677Z [INFO]  server-2c7fd39a-2e10-b227-9700-82e35428c82b: entering follower state: follower="Node at 2c7fd39a-2e10-b227-9700-82e35428c82b [Follower]" leader-address= leader-id=
2025-06-26T05:12:59.677Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: election won: term=4 tally=2
2025-06-26T05:12:59.677Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: entering leader state: leader="Node at 49d94997-6610-92f1-c1c3-a0e48f2e4492 [Leader]"
2025-06-26T05:12:59.677Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: added peer, starting replication: peer=server-2c7fd39a-2e10-b227-9700-82e35428c82b
2025-06-26T05:12:59.677Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: added peer, starting replication: peer=server-42f4a127-9a79-4785-8778-8e9e20e48587
2025-06-26T05:12:59.677Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to appendEntries to: peer="{Voter server-42f4a127-9a79-4785-8778-8e9e20e48587 42f4a127-9a79-4785-8778-8e9e20e48587}" error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587"
2025-06-26T05:12:59.677Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: pipelining replication: peer="{Voter server-2c7fd39a-2e10-b227-9700-82e35428c82b 2c7fd39a-2e10-b227-9700-82e35428c82b}"
2025-06-26T05:12:59.689Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to heartbeat to: peer=42f4a127-9a79-4785-8778-8e9e20e48587 backoff time=10ms error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587"
2025-06-26T05:12:59.701Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to appendEntries to: peer="{Voter server-42f4a127-9a79-4785-8778-8e9e20e48587 42f4a127-9a79-4785-8778-8e9e20e48587}" error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587"
2025-06-26T05:12:59.713Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to heartbeat to: peer=42f4a127-9a79-4785-8778-8e9e20e48587 backoff time=10ms error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587"
2025-06-26T05:12:59.721Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to appendEntries to: peer="{Voter server-42f4a127-9a79-4785-8778-8e9e20e48587 42f4a127-9a79-4785-8778-8e9e20e48587}" error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587"
2025-06-26T05:12:59.729Z [WARN]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to contact: server-id=server-42f4a127-9a79-4785-8778-8e9e20e48587 time=51.82744ms
2025-06-26T05:12:59.733Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to heartbeat to: peer=42f4a127-9a79-4785-8778-8e9e20e48587 backoff time=10ms error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587"
2025-06-26T05:12:59.737Z [WARN]  server-42f4a127-9a79-4785-8778-8e9e20e48587: Election timeout reached, restarting election
2025-06-26T05:12:59.737Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: entering candidate state: node="Node at 42f4a127-9a79-4785-8778-8e9e20e48587 [Candidate]" term=4
2025-06-26T05:12:59.737Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to make requestVote RPC: target="{Voter server-49d94997-6610-92f1-c1c3-a0e48f2e4492 49d94997-6610-92f1-c1c3-a0e48f2e4492}" error="failed to connect to peer: 49d94997-6610-92f1-c1c3-a0e48f2e4492" term=4
2025-06-26T05:12:59.737Z [ERROR] server-42f4a127-9a79-4785-8778-8e9e20e48587: failed to make requestVote RPC: target="{Voter server-2c7fd39a-2e10-b227-9700-82e35428c82b 2c7fd39a-2e10-b227-9700-82e35428c82b}" error="failed to connect to peer: 2c7fd39a-2e10-b227-9700-82e35428c82b" term=4
2025-06-26T05:12:59.749Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to appendEntries to: peer="{Voter server-42f4a127-9a79-4785-8778-8e9e20e48587 42f4a127-9a79-4785-8778-8e9e20e48587}" error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587"
2025-06-26T05:12:59.753Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to heartbeat to: peer=42f4a127-9a79-4785-8778-8e9e20e48587 backoff time=20ms error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587"
2025-06-26T05:12:59.777Z [WARN]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to contact: server-id=server-42f4a127-9a79-4785-8778-8e9e20e48587 time=99.75908ms
2025-06-26T05:12:59.781Z [ERROR] server-49d94997-6610-92f1-c1c3-a0e48f2e4492: failed to heartbeat to: peer=42f4a127-9a79-4785-8778-8e9e20e48587 backoff time=25ms error="failed to connect to peer: 42f4a127-9a79-4785-8778-8e9e20e48587"
2025-06-26T05:12:59.785Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:59.677917603 +0000 UTC m=+7.827027841 (1 nodes), 144.26628ms from start of poll, 443.92632ms from cluster start. Timeout at 2025-06-26 05:12:59.785601163 +0000 UTC m=+7.934711441, 107.6836ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at 42f4a127-9a79-4785-8778-8e9e20e48587 [Candidate]
2025-06-26T05:12:59.805Z [INFO]  server-42f4a127-9a79-4785-8778-8e9e20e48587: entering follower state: follower="Node at 42f4a127-9a79-4785-8778-8e9e20e48587 [Follower]" leader-address=49d94997-6610-92f1-c1c3-a0e48f2e4492 leader-id=server-49d94997-6610-92f1-c1c3-a0e48f2e4492
2025-06-26T05:12:59.805Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: pipelining replication: peer="{Voter server-42f4a127-9a79-4785-8778-8e9e20e48587 42f4a127-9a79-4785-8778-8e9e20e48587}"
2025-06-26T05:12:59.805Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: aborting pipeline replication: peer="{Voter server-42f4a127-9a79-4785-8778-8e9e20e48587 42f4a127-9a79-4785-8778-8e9e20e48587}"
2025-06-26T05:12:59.805Z [INFO]  server-49d94997-6610-92f1-c1c3-a0e48f2e4492: aborting pipeline replication: peer="{Voter server-2c7fd39a-2e10-b227-9700-82e35428c82b 2c7fd39a-2e10-b227-9700-82e35428c82b}"
--- PASS: TestRaft_LeaderFail (0.57s)
=== RUN   TestRaft_BehindFollower
2025-06-26T05:12:59.806Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8 Address:3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8} {Suffrage:Voter ID:server-d904184f-20af-cecd-e11d-3e7b8841076a Address:d904184f-20af-cecd-e11d-3e7b8841076a} {Suffrage:Voter ID:server-b708be2e-a14e-316e-885b-e2587b1b1122 Address:b708be2e-a14e-316e-885b-e2587b1b1122}]"
2025-06-26T05:12:59.806Z [INFO]  server-d904184f-20af-cecd-e11d-3e7b8841076a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8 Address:3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8} {Suffrage:Voter ID:server-d904184f-20af-cecd-e11d-3e7b8841076a Address:d904184f-20af-cecd-e11d-3e7b8841076a} {Suffrage:Voter ID:server-b708be2e-a14e-316e-885b-e2587b1b1122 Address:b708be2e-a14e-316e-885b-e2587b1b1122}]"
2025-06-26T05:12:59.806Z [INFO]  server-b708be2e-a14e-316e-885b-e2587b1b1122: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8 Address:3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8} {Suffrage:Voter ID:server-d904184f-20af-cecd-e11d-3e7b8841076a Address:d904184f-20af-cecd-e11d-3e7b8841076a} {Suffrage:Voter ID:server-b708be2e-a14e-316e-885b-e2587b1b1122 Address:b708be2e-a14e-316e-885b-e2587b1b1122}]"
2025-06-26T05:12:59.806Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:12:59.806Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: entering follower state: follower="Node at 3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8 [Follower]" leader-address= leader-id=
2025-06-26T05:12:59.806Z [INFO]  server-d904184f-20af-cecd-e11d-3e7b8841076a: entering follower state: follower="Node at d904184f-20af-cecd-e11d-3e7b8841076a [Follower]" leader-address= leader-id=
2025-06-26T05:12:59.806Z [INFO]  server-b708be2e-a14e-316e-885b-e2587b1b1122: entering follower state: follower="Node at b708be2e-a14e-316e-885b-e2587b1b1122 [Follower]" leader-address= leader-id=
2025-06-26T05:12:59.865Z [WARN]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:12:59.865Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: entering candidate state: node="Node at 3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8 [Candidate]" term=2
2025-06-26T05:12:59.865Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: election won: term=2 tally=2
2025-06-26T05:12:59.865Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: entering leader state: leader="Node at 3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8 [Leader]"
2025-06-26T05:12:59.865Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: added peer, starting replication: peer=server-d904184f-20af-cecd-e11d-3e7b8841076a
2025-06-26T05:12:59.865Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: added peer, starting replication: peer=server-b708be2e-a14e-316e-885b-e2587b1b1122
2025-06-26T05:12:59.865Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: pipelining replication: peer="{Voter server-b708be2e-a14e-316e-885b-e2587b1b1122 b708be2e-a14e-316e-885b-e2587b1b1122}"
2025-06-26T05:12:59.865Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: pipelining replication: peer="{Voter server-d904184f-20af-cecd-e11d-3e7b8841076a d904184f-20af-cecd-e11d-3e7b8841076a}"
2025-06-26T05:12:59.977Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:12:59.866010803 +0000 UTC m=+8.015121081 (1 nodes), 59.525ms from start of poll, 59.7224ms from cluster start. Timeout at 2025-06-26 05:12:59.977601083 +0000 UTC m=+8.126711361, 111.59028ms after stability
2025-06-26T05:12:59.977Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:00.089Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:12:59.977672963 +0000 UTC m=+8.126783241 (2 nodes), 1.96µs from start of poll, 171.38456ms from cluster start. Timeout at 2025-06-26 05:13:00.089595483 +0000 UTC m=+8.238705761, 111.92252ms after stability
2025-06-26T05:13:00.089Z [ERROR] server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: failed to pipeline appendEntries: peer="{Voter server-d904184f-20af-cecd-e11d-3e7b8841076a d904184f-20af-cecd-e11d-3e7b8841076a}" error="append pipeline closed"
2025-06-26T05:13:00.089Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: aborting pipeline replication: peer="{Voter server-d904184f-20af-cecd-e11d-3e7b8841076a d904184f-20af-cecd-e11d-3e7b8841076a}"
2025-06-26T05:13:00.089Z [ERROR] server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: failed to appendEntries to: peer="{Voter server-d904184f-20af-cecd-e11d-3e7b8841076a d904184f-20af-cecd-e11d-3e7b8841076a}" error="failed to connect to peer: d904184f-20af-cecd-e11d-3e7b8841076a"
2025-06-26T05:13:00.101Z [ERROR] server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: failed to heartbeat to: peer=d904184f-20af-cecd-e11d-3e7b8841076a backoff time=10ms error="failed to connect to peer: d904184f-20af-cecd-e11d-3e7b8841076a"
2025-06-26T05:13:00.101Z [ERROR] server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: failed to appendEntries to: peer="{Voter server-d904184f-20af-cecd-e11d-3e7b8841076a d904184f-20af-cecd-e11d-3e7b8841076a}" error="failed to connect to peer: d904184f-20af-cecd-e11d-3e7b8841076a"
2025-06-26T05:13:00.117Z [ERROR] server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: failed to appendEntries to: peer="{Voter server-d904184f-20af-cecd-e11d-3e7b8841076a d904184f-20af-cecd-e11d-3e7b8841076a}" error="failed to connect to peer: d904184f-20af-cecd-e11d-3e7b8841076a"
    raft_test.go:459: [INFO] Finished apply without behind follower
2025-06-26T05:13:00.153Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: pipelining replication: peer="{Voter server-d904184f-20af-cecd-e11d-3e7b8841076a d904184f-20af-cecd-e11d-3e7b8841076a}"
2025-06-26T05:13:00.169Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:00.277Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:00.169692443 +0000 UTC m=+8.318802721 (1 nodes), 3.04µs from start of poll, 363.40404ms from cluster start. Timeout at 2025-06-26 05:13:00.277620563 +0000 UTC m=+8.426730801, 107.92808ms after stability
2025-06-26T05:13:00.277Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: aborting pipeline replication: peer="{Voter server-b708be2e-a14e-316e-885b-e2587b1b1122 b708be2e-a14e-316e-885b-e2587b1b1122}"
2025-06-26T05:13:00.277Z [INFO]  server-3cd5132c-50a9-4a50-1ba3-7ff1c81eacc8: aborting pipeline replication: peer="{Voter server-d904184f-20af-cecd-e11d-3e7b8841076a d904184f-20af-cecd-e11d-3e7b8841076a}"
--- PASS: TestRaft_BehindFollower (0.47s)
=== RUN   TestRaft_ApplyNonLeader
2025-06-26T05:13:00.278Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1ac7a85-2940-f53c-2cd2-99820c30a501 Address:f1ac7a85-2940-f53c-2cd2-99820c30a501} {Suffrage:Voter ID:server-e69cea73-b543-0be3-e134-3b8ed2d438fe Address:e69cea73-b543-0be3-e134-3b8ed2d438fe} {Suffrage:Voter ID:server-e9546d1c-3b0f-3886-6e49-e9dc90030e25 Address:e9546d1c-3b0f-3886-6e49-e9dc90030e25}]"
2025-06-26T05:13:00.278Z [INFO]  server-e69cea73-b543-0be3-e134-3b8ed2d438fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1ac7a85-2940-f53c-2cd2-99820c30a501 Address:f1ac7a85-2940-f53c-2cd2-99820c30a501} {Suffrage:Voter ID:server-e69cea73-b543-0be3-e134-3b8ed2d438fe Address:e69cea73-b543-0be3-e134-3b8ed2d438fe} {Suffrage:Voter ID:server-e9546d1c-3b0f-3886-6e49-e9dc90030e25 Address:e9546d1c-3b0f-3886-6e49-e9dc90030e25}]"
2025-06-26T05:13:00.278Z [INFO]  server-e9546d1c-3b0f-3886-6e49-e9dc90030e25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1ac7a85-2940-f53c-2cd2-99820c30a501 Address:f1ac7a85-2940-f53c-2cd2-99820c30a501} {Suffrage:Voter ID:server-e69cea73-b543-0be3-e134-3b8ed2d438fe Address:e69cea73-b543-0be3-e134-3b8ed2d438fe} {Suffrage:Voter ID:server-e9546d1c-3b0f-3886-6e49-e9dc90030e25 Address:e9546d1c-3b0f-3886-6e49-e9dc90030e25}]"
2025-06-26T05:13:00.278Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:00.278Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: entering follower state: follower="Node at f1ac7a85-2940-f53c-2cd2-99820c30a501 [Follower]" leader-address= leader-id=
2025-06-26T05:13:00.278Z [INFO]  server-e69cea73-b543-0be3-e134-3b8ed2d438fe: entering follower state: follower="Node at e69cea73-b543-0be3-e134-3b8ed2d438fe [Follower]" leader-address= leader-id=
2025-06-26T05:13:00.278Z [INFO]  server-e9546d1c-3b0f-3886-6e49-e9dc90030e25: entering follower state: follower="Node at e9546d1c-3b0f-3886-6e49-e9dc90030e25 [Follower]" leader-address= leader-id=
2025-06-26T05:13:00.345Z [WARN]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:00.345Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: entering candidate state: node="Node at f1ac7a85-2940-f53c-2cd2-99820c30a501 [Candidate]" term=2
2025-06-26T05:13:00.345Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: election won: term=2 tally=2
2025-06-26T05:13:00.345Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: entering leader state: leader="Node at f1ac7a85-2940-f53c-2cd2-99820c30a501 [Leader]"
2025-06-26T05:13:00.345Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: added peer, starting replication: peer=server-e69cea73-b543-0be3-e134-3b8ed2d438fe
2025-06-26T05:13:00.345Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: added peer, starting replication: peer=server-e9546d1c-3b0f-3886-6e49-e9dc90030e25
2025-06-26T05:13:00.345Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: pipelining replication: peer="{Voter server-e9546d1c-3b0f-3886-6e49-e9dc90030e25 e9546d1c-3b0f-3886-6e49-e9dc90030e25}"
2025-06-26T05:13:00.345Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: pipelining replication: peer="{Voter server-e69cea73-b543-0be3-e134-3b8ed2d438fe e69cea73-b543-0be3-e134-3b8ed2d438fe}"
2025-06-26T05:13:00.453Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:00.346005563 +0000 UTC m=+8.495115841 (1 nodes), 67.72396ms from start of poll, 67.91188ms from cluster start. Timeout at 2025-06-26 05:13:00.453566683 +0000 UTC m=+8.602676961, 107.56112ms after stability
2025-06-26T05:13:00.453Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:00.561Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:00.453623043 +0000 UTC m=+8.602733321 (2 nodes), 1.64µs from start of poll, 175.52936ms from cluster start. Timeout at 2025-06-26 05:13:00.561601043 +0000 UTC m=+8.710711321, 107.978ms after stability
2025-06-26T05:13:00.561Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: aborting pipeline replication: peer="{Voter server-e9546d1c-3b0f-3886-6e49-e9dc90030e25 e9546d1c-3b0f-3886-6e49-e9dc90030e25}"
2025-06-26T05:13:00.565Z [INFO]  server-f1ac7a85-2940-f53c-2cd2-99820c30a501: aborting pipeline replication: peer="{Voter server-e69cea73-b543-0be3-e134-3b8ed2d438fe e69cea73-b543-0be3-e134-3b8ed2d438fe}"
--- PASS: TestRaft_ApplyNonLeader (0.29s)
=== RUN   TestRaft_ApplyConcurrent
2025-06-26T05:13:00.566Z [INFO]  server-73e917ec-5ee9-cc15-d1b9-9eea14afc3b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73e917ec-5ee9-cc15-d1b9-9eea14afc3b6 Address:73e917ec-5ee9-cc15-d1b9-9eea14afc3b6} {Suffrage:Voter ID:server-ec36af63-7923-aa0d-62be-7b8fc7e7a181 Address:ec36af63-7923-aa0d-62be-7b8fc7e7a181} {Suffrage:Voter ID:server-c9084b31-b529-82cf-edb5-e1042bce9530 Address:c9084b31-b529-82cf-edb5-e1042bce9530}]"
2025-06-26T05:13:00.566Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73e917ec-5ee9-cc15-d1b9-9eea14afc3b6 Address:73e917ec-5ee9-cc15-d1b9-9eea14afc3b6} {Suffrage:Voter ID:server-ec36af63-7923-aa0d-62be-7b8fc7e7a181 Address:ec36af63-7923-aa0d-62be-7b8fc7e7a181} {Suffrage:Voter ID:server-c9084b31-b529-82cf-edb5-e1042bce9530 Address:c9084b31-b529-82cf-edb5-e1042bce9530}]"
2025-06-26T05:13:00.566Z [INFO]  server-c9084b31-b529-82cf-edb5-e1042bce9530: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73e917ec-5ee9-cc15-d1b9-9eea14afc3b6 Address:73e917ec-5ee9-cc15-d1b9-9eea14afc3b6} {Suffrage:Voter ID:server-ec36af63-7923-aa0d-62be-7b8fc7e7a181 Address:ec36af63-7923-aa0d-62be-7b8fc7e7a181} {Suffrage:Voter ID:server-c9084b31-b529-82cf-edb5-e1042bce9530 Address:c9084b31-b529-82cf-edb5-e1042bce9530}]"
2025-06-26T05:13:00.566Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:00.566Z [INFO]  server-73e917ec-5ee9-cc15-d1b9-9eea14afc3b6: entering follower state: follower="Node at 73e917ec-5ee9-cc15-d1b9-9eea14afc3b6 [Follower]" leader-address= leader-id=
2025-06-26T05:13:00.566Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: entering follower state: follower="Node at ec36af63-7923-aa0d-62be-7b8fc7e7a181 [Follower]" leader-address= leader-id=
2025-06-26T05:13:00.566Z [INFO]  server-c9084b31-b529-82cf-edb5-e1042bce9530: entering follower state: follower="Node at c9084b31-b529-82cf-edb5-e1042bce9530 [Follower]" leader-address= leader-id=
2025-06-26T05:13:00.669Z [WARN]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:00.669Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: entering candidate state: node="Node at ec36af63-7923-aa0d-62be-7b8fc7e7a181 [Candidate]" term=2
2025-06-26T05:13:00.669Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: election won: term=2 tally=2
2025-06-26T05:13:00.669Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: entering leader state: leader="Node at ec36af63-7923-aa0d-62be-7b8fc7e7a181 [Leader]"
2025-06-26T05:13:00.669Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: added peer, starting replication: peer=server-73e917ec-5ee9-cc15-d1b9-9eea14afc3b6
2025-06-26T05:13:00.669Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: added peer, starting replication: peer=server-c9084b31-b529-82cf-edb5-e1042bce9530
2025-06-26T05:13:00.669Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: pipelining replication: peer="{Voter server-c9084b31-b529-82cf-edb5-e1042bce9530 c9084b31-b529-82cf-edb5-e1042bce9530}"
2025-06-26T05:13:00.669Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: pipelining replication: peer="{Voter server-73e917ec-5ee9-cc15-d1b9-9eea14afc3b6 73e917ec-5ee9-cc15-d1b9-9eea14afc3b6}"
2025-06-26T05:13:00.877Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:00.669957523 +0000 UTC m=+8.819067801 (1 nodes), 103.67508ms from start of poll, 103.90716ms from cluster start. Timeout at 2025-06-26 05:13:00.877676403 +0000 UTC m=+9.026786681, 207.71888ms after stability
2025-06-26T05:13:00.921Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: aborting pipeline replication: peer="{Voter server-c9084b31-b529-82cf-edb5-e1042bce9530 c9084b31-b529-82cf-edb5-e1042bce9530}"
2025-06-26T05:13:00.922Z [INFO]  server-ec36af63-7923-aa0d-62be-7b8fc7e7a181: aborting pipeline replication: peer="{Voter server-73e917ec-5ee9-cc15-d1b9-9eea14afc3b6 73e917ec-5ee9-cc15-d1b9-9eea14afc3b6}"
--- PASS: TestRaft_ApplyConcurrent (0.36s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2025-06-26T05:13:00.922Z [INFO]  server-778147a5-e729-9f75-45bf-2a4562d5bff0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-778147a5-e729-9f75-45bf-2a4562d5bff0 Address:778147a5-e729-9f75-45bf-2a4562d5bff0}]"
2025-06-26T05:13:00.922Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:00.922Z [INFO]  server-778147a5-e729-9f75-45bf-2a4562d5bff0: entering follower state: follower="Node at 778147a5-e729-9f75-45bf-2a4562d5bff0 [Follower]" leader-address= leader-id=
2025-06-26T05:13:01.089Z [WARN]  server-778147a5-e729-9f75-45bf-2a4562d5bff0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:01.089Z [INFO]  server-778147a5-e729-9f75-45bf-2a4562d5bff0: entering candidate state: node="Node at 778147a5-e729-9f75-45bf-2a4562d5bff0 [Candidate]" term=2
2025-06-26T05:13:01.089Z [INFO]  server-778147a5-e729-9f75-45bf-2a4562d5bff0: election won: term=2 tally=1
2025-06-26T05:13:01.089Z [INFO]  server-778147a5-e729-9f75-45bf-2a4562d5bff0: entering leader state: leader="Node at 778147a5-e729-9f75-45bf-2a4562d5bff0 [Leader]"
2025-06-26T05:13:01.293Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:01.089766363 +0000 UTC m=+9.238876641 (1 nodes), 167.34016ms from start of poll, 167.4346ms from cluster start. Timeout at 2025-06-26 05:13:01.293603723 +0000 UTC m=+9.442714001, 203.83736ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.37s)
=== RUN   TestRaft_JoinNode
2025-06-26T05:13:01.296Z [INFO]  server-e29c45ac-2d7d-7ff7-bd59-f92b67547fd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e29c45ac-2d7d-7ff7-bd59-f92b67547fd6 Address:e29c45ac-2d7d-7ff7-bd59-f92b67547fd6} {Suffrage:Voter ID:server-7b7db304-1df6-a551-9f4a-6427a962e75b Address:7b7db304-1df6-a551-9f4a-6427a962e75b}]"
2025-06-26T05:13:01.296Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e29c45ac-2d7d-7ff7-bd59-f92b67547fd6 Address:e29c45ac-2d7d-7ff7-bd59-f92b67547fd6} {Suffrage:Voter ID:server-7b7db304-1df6-a551-9f4a-6427a962e75b Address:7b7db304-1df6-a551-9f4a-6427a962e75b}]"
2025-06-26T05:13:01.296Z [INFO]  server-70df3593-d472-5d8a-a905-e59d99625de7: initial configuration: index=0 servers=[]
2025-06-26T05:13:01.296Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:01.296Z [INFO]  server-e29c45ac-2d7d-7ff7-bd59-f92b67547fd6: entering follower state: follower="Node at e29c45ac-2d7d-7ff7-bd59-f92b67547fd6 [Follower]" leader-address= leader-id=
2025-06-26T05:13:01.296Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: entering follower state: follower="Node at 7b7db304-1df6-a551-9f4a-6427a962e75b [Follower]" leader-address= leader-id=
2025-06-26T05:13:01.296Z [INFO]  server-70df3593-d472-5d8a-a905-e59d99625de7: entering follower state: follower="Node at 70df3593-d472-5d8a-a905-e59d99625de7 [Follower]" leader-address= leader-id=
2025-06-26T05:13:01.361Z [WARN]  server-70df3593-d472-5d8a-a905-e59d99625de7: no known peers, aborting election
2025-06-26T05:13:01.365Z [WARN]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:01.365Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: entering candidate state: node="Node at 7b7db304-1df6-a551-9f4a-6427a962e75b [Candidate]" term=2
2025-06-26T05:13:01.365Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: election won: term=2 tally=2
2025-06-26T05:13:01.365Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: entering leader state: leader="Node at 7b7db304-1df6-a551-9f4a-6427a962e75b [Leader]"
2025-06-26T05:13:01.365Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: added peer, starting replication: peer=server-e29c45ac-2d7d-7ff7-bd59-f92b67547fd6
2025-06-26T05:13:01.365Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: pipelining replication: peer="{Voter server-e29c45ac-2d7d-7ff7-bd59-f92b67547fd6 e29c45ac-2d7d-7ff7-bd59-f92b67547fd6}"
2025-06-26T05:13:01.473Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:01.365844523 +0000 UTC m=+9.514954801 (1 nodes), 69.3976ms from start of poll, 69.61752ms from cluster start. Timeout at 2025-06-26 05:13:01.473574603 +0000 UTC m=+9.622684881, 107.73008ms after stability
2025-06-26T05:13:01.473Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: updating configuration: command=AddVoter server-id=server-70df3593-d472-5d8a-a905-e59d99625de7 server-addr=70df3593-d472-5d8a-a905-e59d99625de7 servers="[{Suffrage:Voter ID:server-e29c45ac-2d7d-7ff7-bd59-f92b67547fd6 Address:e29c45ac-2d7d-7ff7-bd59-f92b67547fd6} {Suffrage:Voter ID:server-7b7db304-1df6-a551-9f4a-6427a962e75b Address:7b7db304-1df6-a551-9f4a-6427a962e75b} {Suffrage:Voter ID:server-70df3593-d472-5d8a-a905-e59d99625de7 Address:70df3593-d472-5d8a-a905-e59d99625de7}]"
2025-06-26T05:13:01.473Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: added peer, starting replication: peer=server-70df3593-d472-5d8a-a905-e59d99625de7
2025-06-26T05:13:01.473Z [WARN]  server-70df3593-d472-5d8a-a905-e59d99625de7: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-06-26T05:13:01.473Z [WARN]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: appendEntries rejected, sending older logs: peer="{Voter server-70df3593-d472-5d8a-a905-e59d99625de7 70df3593-d472-5d8a-a905-e59d99625de7}" next=1
2025-06-26T05:13:01.473Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: pipelining replication: peer="{Voter server-70df3593-d472-5d8a-a905-e59d99625de7 70df3593-d472-5d8a-a905-e59d99625de7}"
2025-06-26T05:13:01.473Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:01.581Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:01.473906283 +0000 UTC m=+9.623016521 (1 nodes), 800ns from start of poll, 177.67924ms from cluster start. Timeout at 2025-06-26 05:13:01.581565763 +0000 UTC m=+9.730676001, 107.65948ms after stability
2025-06-26T05:13:01.581Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: aborting pipeline replication: peer="{Voter server-70df3593-d472-5d8a-a905-e59d99625de7 70df3593-d472-5d8a-a905-e59d99625de7}"
2025-06-26T05:13:01.581Z [INFO]  server-7b7db304-1df6-a551-9f4a-6427a962e75b: aborting pipeline replication: peer="{Voter server-e29c45ac-2d7d-7ff7-bd59-f92b67547fd6 e29c45ac-2d7d-7ff7-bd59-f92b67547fd6}"
--- PASS: TestRaft_JoinNode (0.29s)
=== RUN   TestRaft_JoinNode_ConfigStore
2025-06-26T05:13:01.582Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98119778-a994-32e6-ed62-5a9e7bf3c487 Address:98119778-a994-32e6-ed62-5a9e7bf3c487}]"
2025-06-26T05:13:01.582Z [INFO]  server-69009c94-1888-4962-5535-84092a72081b: initial configuration: index=0 servers=[]
2025-06-26T05:13:01.582Z [INFO]  server-5447f862-7a2d-3733-1349-815633c0f1b6: initial configuration: index=0 servers=[]
2025-06-26T05:13:01.582Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:01.582Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: entering follower state: follower="Node at 98119778-a994-32e6-ed62-5a9e7bf3c487 [Follower]" leader-address= leader-id=
2025-06-26T05:13:01.582Z [INFO]  server-69009c94-1888-4962-5535-84092a72081b: entering follower state: follower="Node at 69009c94-1888-4962-5535-84092a72081b [Follower]" leader-address= leader-id=
2025-06-26T05:13:01.582Z [INFO]  server-5447f862-7a2d-3733-1349-815633c0f1b6: entering follower state: follower="Node at 5447f862-7a2d-3733-1349-815633c0f1b6 [Follower]" leader-address= leader-id=
2025-06-26T05:13:01.633Z [WARN]  server-5447f862-7a2d-3733-1349-815633c0f1b6: no known peers, aborting election
2025-06-26T05:13:01.633Z [WARN]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:01.633Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: entering candidate state: node="Node at 98119778-a994-32e6-ed62-5a9e7bf3c487 [Candidate]" term=2
2025-06-26T05:13:01.633Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: election won: term=2 tally=1
2025-06-26T05:13:01.633Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: entering leader state: leader="Node at 98119778-a994-32e6-ed62-5a9e7bf3c487 [Leader]"
2025-06-26T05:13:01.645Z [WARN]  server-69009c94-1888-4962-5535-84092a72081b: no known peers, aborting election
2025-06-26T05:13:01.741Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:01.633786363 +0000 UTC m=+9.782896641 (1 nodes), 51.51016ms from start of poll, 51.74384ms from cluster start. Timeout at 2025-06-26 05:13:01.741570283 +0000 UTC m=+9.890680561, 107.78392ms after stability
2025-06-26T05:13:01.741Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: updating configuration: command=AddVoter server-id=server-69009c94-1888-4962-5535-84092a72081b server-addr=69009c94-1888-4962-5535-84092a72081b servers="[{Suffrage:Voter ID:server-98119778-a994-32e6-ed62-5a9e7bf3c487 Address:98119778-a994-32e6-ed62-5a9e7bf3c487} {Suffrage:Voter ID:server-69009c94-1888-4962-5535-84092a72081b Address:69009c94-1888-4962-5535-84092a72081b}]"
2025-06-26T05:13:01.741Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: added peer, starting replication: peer=server-69009c94-1888-4962-5535-84092a72081b
2025-06-26T05:13:01.741Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:01.741Z [WARN]  server-69009c94-1888-4962-5535-84092a72081b: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-06-26T05:13:01.741Z [WARN]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: appendEntries rejected, sending older logs: peer="{Voter server-69009c94-1888-4962-5535-84092a72081b 69009c94-1888-4962-5535-84092a72081b}" next=1
2025-06-26T05:13:01.741Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: pipelining replication: peer="{Voter server-69009c94-1888-4962-5535-84092a72081b 69009c94-1888-4962-5535-84092a72081b}"
2025-06-26T05:13:01.849Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:01.741758203 +0000 UTC m=+9.890868481 (1 nodes), 560ns from start of poll, 159.71568ms from cluster start. Timeout at 2025-06-26 05:13:01.849595163 +0000 UTC m=+9.998705441, 107.83696ms after stability
2025-06-26T05:13:01.849Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: updating configuration: command=AddVoter server-id=server-5447f862-7a2d-3733-1349-815633c0f1b6 server-addr=5447f862-7a2d-3733-1349-815633c0f1b6 servers="[{Suffrage:Voter ID:server-98119778-a994-32e6-ed62-5a9e7bf3c487 Address:98119778-a994-32e6-ed62-5a9e7bf3c487} {Suffrage:Voter ID:server-69009c94-1888-4962-5535-84092a72081b Address:69009c94-1888-4962-5535-84092a72081b} {Suffrage:Voter ID:server-5447f862-7a2d-3733-1349-815633c0f1b6 Address:5447f862-7a2d-3733-1349-815633c0f1b6}]"
2025-06-26T05:13:01.849Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: added peer, starting replication: peer=server-5447f862-7a2d-3733-1349-815633c0f1b6
2025-06-26T05:13:01.849Z [WARN]  server-5447f862-7a2d-3733-1349-815633c0f1b6: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2025-06-26T05:13:01.849Z [WARN]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: appendEntries rejected, sending older logs: peer="{Voter server-5447f862-7a2d-3733-1349-815633c0f1b6 5447f862-7a2d-3733-1349-815633c0f1b6}" next=1
2025-06-26T05:13:01.849Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: pipelining replication: peer="{Voter server-5447f862-7a2d-3733-1349-815633c0f1b6 5447f862-7a2d-3733-1349-815633c0f1b6}"
2025-06-26T05:13:01.849Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:01.957Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:01.849928963 +0000 UTC m=+9.999039241 (1 nodes), 720ns from start of poll, 267.88644ms from cluster start. Timeout at 2025-06-26 05:13:01.957627523 +0000 UTC m=+10.106737801, 107.69856ms after stability
2025-06-26T05:13:01.957Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: aborting pipeline replication: peer="{Voter server-5447f862-7a2d-3733-1349-815633c0f1b6 5447f862-7a2d-3733-1349-815633c0f1b6}"
2025-06-26T05:13:01.957Z [INFO]  server-98119778-a994-32e6-ed62-5a9e7bf3c487: aborting pipeline replication: peer="{Voter server-69009c94-1888-4962-5535-84092a72081b 69009c94-1888-4962-5535-84092a72081b}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.38s)
=== RUN   TestRaft_RemoveFollower
2025-06-26T05:13:01.958Z [INFO]  server-d229852c-9886-0d9a-7a80-17222ec8c44b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d229852c-9886-0d9a-7a80-17222ec8c44b Address:d229852c-9886-0d9a-7a80-17222ec8c44b} {Suffrage:Voter ID:server-7528876b-917a-959c-fe7a-e124c7a0bd66 Address:7528876b-917a-959c-fe7a-e124c7a0bd66} {Suffrage:Voter ID:server-2598b466-551a-f595-eeea-ae69ea265060 Address:2598b466-551a-f595-eeea-ae69ea265060}]"
2025-06-26T05:13:01.958Z [INFO]  server-7528876b-917a-959c-fe7a-e124c7a0bd66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d229852c-9886-0d9a-7a80-17222ec8c44b Address:d229852c-9886-0d9a-7a80-17222ec8c44b} {Suffrage:Voter ID:server-7528876b-917a-959c-fe7a-e124c7a0bd66 Address:7528876b-917a-959c-fe7a-e124c7a0bd66} {Suffrage:Voter ID:server-2598b466-551a-f595-eeea-ae69ea265060 Address:2598b466-551a-f595-eeea-ae69ea265060}]"
2025-06-26T05:13:01.958Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d229852c-9886-0d9a-7a80-17222ec8c44b Address:d229852c-9886-0d9a-7a80-17222ec8c44b} {Suffrage:Voter ID:server-7528876b-917a-959c-fe7a-e124c7a0bd66 Address:7528876b-917a-959c-fe7a-e124c7a0bd66} {Suffrage:Voter ID:server-2598b466-551a-f595-eeea-ae69ea265060 Address:2598b466-551a-f595-eeea-ae69ea265060}]"
2025-06-26T05:13:01.958Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:01.958Z [INFO]  server-d229852c-9886-0d9a-7a80-17222ec8c44b: entering follower state: follower="Node at d229852c-9886-0d9a-7a80-17222ec8c44b [Follower]" leader-address= leader-id=
2025-06-26T05:13:01.958Z [INFO]  server-7528876b-917a-959c-fe7a-e124c7a0bd66: entering follower state: follower="Node at 7528876b-917a-959c-fe7a-e124c7a0bd66 [Follower]" leader-address= leader-id=
2025-06-26T05:13:01.958Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: entering follower state: follower="Node at 2598b466-551a-f595-eeea-ae69ea265060 [Follower]" leader-address= leader-id=
2025-06-26T05:13:02.017Z [WARN]  server-2598b466-551a-f595-eeea-ae69ea265060: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:02.017Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: entering candidate state: node="Node at 2598b466-551a-f595-eeea-ae69ea265060 [Candidate]" term=2
2025-06-26T05:13:02.017Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: election won: term=2 tally=2
2025-06-26T05:13:02.017Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: entering leader state: leader="Node at 2598b466-551a-f595-eeea-ae69ea265060 [Leader]"
2025-06-26T05:13:02.017Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: added peer, starting replication: peer=server-d229852c-9886-0d9a-7a80-17222ec8c44b
2025-06-26T05:13:02.017Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: added peer, starting replication: peer=server-7528876b-917a-959c-fe7a-e124c7a0bd66
2025-06-26T05:13:02.017Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: pipelining replication: peer="{Voter server-7528876b-917a-959c-fe7a-e124c7a0bd66 7528876b-917a-959c-fe7a-e124c7a0bd66}"
2025-06-26T05:13:02.017Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: pipelining replication: peer="{Voter server-d229852c-9886-0d9a-7a80-17222ec8c44b d229852c-9886-0d9a-7a80-17222ec8c44b}"
2025-06-26T05:13:02.125Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:02.017988643 +0000 UTC m=+10.167098921 (1 nodes), 59.69392ms from start of poll, 59.87524ms from cluster start. Timeout at 2025-06-26 05:13:02.125597283 +0000 UTC m=+10.274707561, 107.60864ms after stability
2025-06-26T05:13:02.133Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:02.241Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:02.133653283 +0000 UTC m=+10.282763561 (2 nodes), 1.2µs from start of poll, 175.53988ms from cluster start. Timeout at 2025-06-26 05:13:02.241600683 +0000 UTC m=+10.390710961, 107.9474ms after stability
2025-06-26T05:13:02.241Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: updating configuration: command=RemoveServer server-id=server-d229852c-9886-0d9a-7a80-17222ec8c44b server-addr= servers="[{Suffrage:Voter ID:server-7528876b-917a-959c-fe7a-e124c7a0bd66 Address:7528876b-917a-959c-fe7a-e124c7a0bd66} {Suffrage:Voter ID:server-2598b466-551a-f595-eeea-ae69ea265060 Address:2598b466-551a-f595-eeea-ae69ea265060}]"
2025-06-26T05:13:02.241Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: removed peer, stopping replication: peer=server-d229852c-9886-0d9a-7a80-17222ec8c44b last-index=3
2025-06-26T05:13:02.241Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: aborting pipeline replication: peer="{Voter server-d229852c-9886-0d9a-7a80-17222ec8c44b d229852c-9886-0d9a-7a80-17222ec8c44b}"
2025-06-26T05:13:02.329Z [WARN]  server-d229852c-9886-0d9a-7a80-17222ec8c44b: not part of stable configuration, aborting election
2025-06-26T05:13:02.349Z [INFO]  server-2598b466-551a-f595-eeea-ae69ea265060: aborting pipeline replication: peer="{Voter server-7528876b-917a-959c-fe7a-e124c7a0bd66 7528876b-917a-959c-fe7a-e124c7a0bd66}"
--- PASS: TestRaft_RemoveFollower (0.39s)
=== RUN   TestRaft_RemoveLeader
2025-06-26T05:13:02.350Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b82fabe8-fd2d-aa10-511e-873963b6605f Address:b82fabe8-fd2d-aa10-511e-873963b6605f} {Suffrage:Voter ID:server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 Address:ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97} {Suffrage:Voter ID:server-06e8406a-96aa-fd65-81c4-23cde1106bcf Address:06e8406a-96aa-fd65-81c4-23cde1106bcf}]"
2025-06-26T05:13:02.350Z [INFO]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b82fabe8-fd2d-aa10-511e-873963b6605f Address:b82fabe8-fd2d-aa10-511e-873963b6605f} {Suffrage:Voter ID:server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 Address:ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97} {Suffrage:Voter ID:server-06e8406a-96aa-fd65-81c4-23cde1106bcf Address:06e8406a-96aa-fd65-81c4-23cde1106bcf}]"
2025-06-26T05:13:02.350Z [INFO]  server-06e8406a-96aa-fd65-81c4-23cde1106bcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b82fabe8-fd2d-aa10-511e-873963b6605f Address:b82fabe8-fd2d-aa10-511e-873963b6605f} {Suffrage:Voter ID:server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 Address:ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97} {Suffrage:Voter ID:server-06e8406a-96aa-fd65-81c4-23cde1106bcf Address:06e8406a-96aa-fd65-81c4-23cde1106bcf}]"
2025-06-26T05:13:02.350Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:02.350Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: entering follower state: follower="Node at b82fabe8-fd2d-aa10-511e-873963b6605f [Follower]" leader-address= leader-id=
2025-06-26T05:13:02.350Z [INFO]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: entering follower state: follower="Node at ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 [Follower]" leader-address= leader-id=
2025-06-26T05:13:02.350Z [INFO]  server-06e8406a-96aa-fd65-81c4-23cde1106bcf: entering follower state: follower="Node at 06e8406a-96aa-fd65-81c4-23cde1106bcf [Follower]" leader-address= leader-id=
2025-06-26T05:13:02.401Z [WARN]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:02.401Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: entering candidate state: node="Node at b82fabe8-fd2d-aa10-511e-873963b6605f [Candidate]" term=2
2025-06-26T05:13:02.401Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: election won: term=2 tally=2
2025-06-26T05:13:02.401Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: entering leader state: leader="Node at b82fabe8-fd2d-aa10-511e-873963b6605f [Leader]"
2025-06-26T05:13:02.401Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: added peer, starting replication: peer=server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97
2025-06-26T05:13:02.401Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: added peer, starting replication: peer=server-06e8406a-96aa-fd65-81c4-23cde1106bcf
2025-06-26T05:13:02.401Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: pipelining replication: peer="{Voter server-06e8406a-96aa-fd65-81c4-23cde1106bcf 06e8406a-96aa-fd65-81c4-23cde1106bcf}"
2025-06-26T05:13:02.401Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: pipelining replication: peer="{Voter server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97}"
2025-06-26T05:13:02.509Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:02.401940523 +0000 UTC m=+10.551050801 (1 nodes), 51.7816ms from start of poll, 51.9788ms from cluster start. Timeout at 2025-06-26 05:13:02.509574443 +0000 UTC m=+10.658684721, 107.63392ms after stability
2025-06-26T05:13:02.517Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:02.633Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:02.517621323 +0000 UTC m=+10.666731601 (2 nodes), 1µs from start of poll, 167.6596ms from cluster start. Timeout at 2025-06-26 05:13:02.633611003 +0000 UTC m=+10.782721281, 115.98968ms after stability
2025-06-26T05:13:02.633Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: updating configuration: command=RemoveServer server-id=server-b82fabe8-fd2d-aa10-511e-873963b6605f server-addr= servers="[{Suffrage:Voter ID:server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 Address:ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97} {Suffrage:Voter ID:server-06e8406a-96aa-fd65-81c4-23cde1106bcf Address:06e8406a-96aa-fd65-81c4-23cde1106bcf}]"
2025-06-26T05:13:02.633Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: removed ourself, shutting down
2025-06-26T05:13:02.633Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: aborting pipeline replication: peer="{Voter server-06e8406a-96aa-fd65-81c4-23cde1106bcf 06e8406a-96aa-fd65-81c4-23cde1106bcf}"
2025-06-26T05:13:02.633Z [INFO]  server-b82fabe8-fd2d-aa10-511e-873963b6605f: aborting pipeline replication: peer="{Voter server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97}"
2025-06-26T05:13:02.717Z [WARN]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: heartbeat timeout reached, starting election: last-leader-addr=b82fabe8-fd2d-aa10-511e-873963b6605f last-leader-id=server-b82fabe8-fd2d-aa10-511e-873963b6605f
2025-06-26T05:13:02.717Z [INFO]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: entering candidate state: node="Node at ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 [Candidate]" term=3
2025-06-26T05:13:02.717Z [WARN]  server-06e8406a-96aa-fd65-81c4-23cde1106bcf: rejecting vote request since we have a leader: from=ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 leader=b82fabe8-fd2d-aa10-511e-873963b6605f leader-id=server-b82fabe8-fd2d-aa10-511e-873963b6605f
2025-06-26T05:13:02.753Z [WARN]  server-06e8406a-96aa-fd65-81c4-23cde1106bcf: heartbeat timeout reached, starting election: last-leader-addr=b82fabe8-fd2d-aa10-511e-873963b6605f last-leader-id=server-b82fabe8-fd2d-aa10-511e-873963b6605f
2025-06-26T05:13:02.753Z [INFO]  server-06e8406a-96aa-fd65-81c4-23cde1106bcf: entering candidate state: node="Node at 06e8406a-96aa-fd65-81c4-23cde1106bcf [Candidate]" term=3
2025-06-26T05:13:02.753Z [INFO]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: duplicate requestVote for same term: term=3
2025-06-26T05:13:02.789Z [WARN]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: Election timeout reached, restarting election
2025-06-26T05:13:02.789Z [INFO]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: entering candidate state: node="Node at ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 [Candidate]" term=4
2025-06-26T05:13:02.789Z [INFO]  server-06e8406a-96aa-fd65-81c4-23cde1106bcf: entering follower state: follower="Node at 06e8406a-96aa-fd65-81c4-23cde1106bcf [Follower]" leader-address= leader-id=
2025-06-26T05:13:02.789Z [INFO]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: election won: term=4 tally=2
2025-06-26T05:13:02.789Z [INFO]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: entering leader state: leader="Node at ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97 [Leader]"
2025-06-26T05:13:02.789Z [INFO]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: added peer, starting replication: peer=server-06e8406a-96aa-fd65-81c4-23cde1106bcf
2025-06-26T05:13:02.789Z [INFO]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: pipelining replication: peer="{Voter server-06e8406a-96aa-fd65-81c4-23cde1106bcf 06e8406a-96aa-fd65-81c4-23cde1106bcf}"
2025-06-26T05:13:02.849Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:02.957Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:02.849693603 +0000 UTC m=+10.998803881 (1 nodes), 36.2µs from start of poll, 499.73188ms from cluster start. Timeout at 2025-06-26 05:13:02.957577283 +0000 UTC m=+11.106687561, 107.88368ms after stability
2025-06-26T05:13:02.957Z [INFO]  server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: aborting pipeline replication: peer="{Voter server-06e8406a-96aa-fd65-81c4-23cde1106bcf 06e8406a-96aa-fd65-81c4-23cde1106bcf}"
2025-06-26T05:13:03.461Z [ERROR] server-ab5ff3b2-dfc8-2351-a4b3-2d2a0fab7c97: failed to heartbeat to: peer=06e8406a-96aa-fd65-81c4-23cde1106bcf backoff time=10ms error="command timed out"
--- PASS: TestRaft_RemoveLeader (1.11s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2025-06-26T05:13:03.462Z [INFO]  server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc Address:6f1067d3-a104-5938-c6e6-4e32b0b5e1fc} {Suffrage:Voter ID:server-84d28f70-2755-a114-800b-d7bc710dbed8 Address:84d28f70-2755-a114-800b-d7bc710dbed8} {Suffrage:Voter ID:server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa Address:425019ae-2d2b-52ce-b2c1-87f889ff8ffa}]"
2025-06-26T05:13:03.462Z [INFO]  server-84d28f70-2755-a114-800b-d7bc710dbed8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc Address:6f1067d3-a104-5938-c6e6-4e32b0b5e1fc} {Suffrage:Voter ID:server-84d28f70-2755-a114-800b-d7bc710dbed8 Address:84d28f70-2755-a114-800b-d7bc710dbed8} {Suffrage:Voter ID:server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa Address:425019ae-2d2b-52ce-b2c1-87f889ff8ffa}]"
2025-06-26T05:13:03.462Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc Address:6f1067d3-a104-5938-c6e6-4e32b0b5e1fc} {Suffrage:Voter ID:server-84d28f70-2755-a114-800b-d7bc710dbed8 Address:84d28f70-2755-a114-800b-d7bc710dbed8} {Suffrage:Voter ID:server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa Address:425019ae-2d2b-52ce-b2c1-87f889ff8ffa}]"
2025-06-26T05:13:03.462Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:03.462Z [INFO]  server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc: entering follower state: follower="Node at 6f1067d3-a104-5938-c6e6-4e32b0b5e1fc [Follower]" leader-address= leader-id=
2025-06-26T05:13:03.462Z [INFO]  server-84d28f70-2755-a114-800b-d7bc710dbed8: entering follower state: follower="Node at 84d28f70-2755-a114-800b-d7bc710dbed8 [Follower]" leader-address= leader-id=
2025-06-26T05:13:03.462Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: entering follower state: follower="Node at 425019ae-2d2b-52ce-b2c1-87f889ff8ffa [Follower]" leader-address= leader-id=
2025-06-26T05:13:03.541Z [WARN]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:03.541Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: entering candidate state: node="Node at 425019ae-2d2b-52ce-b2c1-87f889ff8ffa [Candidate]" term=2
2025-06-26T05:13:03.541Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: election won: term=2 tally=2
2025-06-26T05:13:03.541Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: entering leader state: leader="Node at 425019ae-2d2b-52ce-b2c1-87f889ff8ffa [Leader]"
2025-06-26T05:13:03.541Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: added peer, starting replication: peer=server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc
2025-06-26T05:13:03.541Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: added peer, starting replication: peer=server-84d28f70-2755-a114-800b-d7bc710dbed8
2025-06-26T05:13:03.541Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: pipelining replication: peer="{Voter server-84d28f70-2755-a114-800b-d7bc710dbed8 84d28f70-2755-a114-800b-d7bc710dbed8}"
2025-06-26T05:13:03.541Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: pipelining replication: peer="{Voter server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc 6f1067d3-a104-5938-c6e6-4e32b0b5e1fc}"
2025-06-26T05:13:03.653Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:03.541927843 +0000 UTC m=+11.691038121 (2 nodes), 79.70996ms from start of poll, 79.90136ms from cluster start. Timeout at 2025-06-26 05:13:03.649627763 +0000 UTC m=+11.798738001, 107.69988ms after stability
2025-06-26T05:13:03.653Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:03.761Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:03.653791443 +0000 UTC m=+11.802901681 (1 nodes), 1.08µs from start of poll, 191.76492ms from cluster start. Timeout at 2025-06-26 05:13:03.761604123 +0000 UTC m=+11.910714401, 107.81272ms after stability
2025-06-26T05:13:03.764Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: updating configuration: command=RemoveServer server-id=server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa server-addr= servers="[{Suffrage:Voter ID:server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc Address:6f1067d3-a104-5938-c6e6-4e32b0b5e1fc} {Suffrage:Voter ID:server-84d28f70-2755-a114-800b-d7bc710dbed8 Address:84d28f70-2755-a114-800b-d7bc710dbed8}]"
2025-06-26T05:13:03.764Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: removed ourself, transitioning to follower
2025-06-26T05:13:03.764Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: entering follower state: follower="Node at 425019ae-2d2b-52ce-b2c1-87f889ff8ffa [Follower]" leader-address= leader-id=
2025-06-26T05:13:03.764Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: aborting pipeline replication: peer="{Voter server-84d28f70-2755-a114-800b-d7bc710dbed8 84d28f70-2755-a114-800b-d7bc710dbed8}"
2025-06-26T05:13:03.764Z [INFO]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: aborting pipeline replication: peer="{Voter server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc 6f1067d3-a104-5938-c6e6-4e32b0b5e1fc}"
2025-06-26T05:13:03.841Z [WARN]  server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa: not part of stable configuration, aborting election
2025-06-26T05:13:03.857Z [WARN]  server-84d28f70-2755-a114-800b-d7bc710dbed8: heartbeat timeout reached, starting election: last-leader-addr=425019ae-2d2b-52ce-b2c1-87f889ff8ffa last-leader-id=server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa
2025-06-26T05:13:03.857Z [INFO]  server-84d28f70-2755-a114-800b-d7bc710dbed8: entering candidate state: node="Node at 84d28f70-2755-a114-800b-d7bc710dbed8 [Candidate]" term=3
2025-06-26T05:13:03.857Z [WARN]  server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc: rejecting vote request since we have a leader: from=84d28f70-2755-a114-800b-d7bc710dbed8 leader=425019ae-2d2b-52ce-b2c1-87f889ff8ffa leader-id=server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa
2025-06-26T05:13:03.869Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:03.881Z [WARN]  server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc: heartbeat timeout reached, starting election: last-leader-addr=425019ae-2d2b-52ce-b2c1-87f889ff8ffa last-leader-id=server-425019ae-2d2b-52ce-b2c1-87f889ff8ffa
2025-06-26T05:13:03.881Z [INFO]  server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc: entering candidate state: node="Node at 6f1067d3-a104-5938-c6e6-4e32b0b5e1fc [Candidate]" term=3
2025-06-26T05:13:03.881Z [INFO]  server-84d28f70-2755-a114-800b-d7bc710dbed8: duplicate requestVote for same term: term=3
2025-06-26T05:13:03.949Z [WARN]  server-84d28f70-2755-a114-800b-d7bc710dbed8: Election timeout reached, restarting election
2025-06-26T05:13:03.949Z [INFO]  server-84d28f70-2755-a114-800b-d7bc710dbed8: entering candidate state: node="Node at 84d28f70-2755-a114-800b-d7bc710dbed8 [Candidate]" term=4
2025-06-26T05:13:03.949Z [INFO]  server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc: entering follower state: follower="Node at 6f1067d3-a104-5938-c6e6-4e32b0b5e1fc [Follower]" leader-address= leader-id=
2025-06-26T05:13:03.949Z [INFO]  server-84d28f70-2755-a114-800b-d7bc710dbed8: election won: term=4 tally=2
2025-06-26T05:13:03.949Z [INFO]  server-84d28f70-2755-a114-800b-d7bc710dbed8: entering leader state: leader="Node at 84d28f70-2755-a114-800b-d7bc710dbed8 [Leader]"
2025-06-26T05:13:03.949Z [INFO]  server-84d28f70-2755-a114-800b-d7bc710dbed8: added peer, starting replication: peer=server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc
2025-06-26T05:13:03.949Z [INFO]  server-84d28f70-2755-a114-800b-d7bc710dbed8: pipelining replication: peer="{Voter server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc 6f1067d3-a104-5938-c6e6-4e32b0b5e1fc}"
2025-06-26T05:13:04.057Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:03.949893283 +0000 UTC m=+12.099003521 (1 nodes), 80.22392ms from start of poll, 487.86676ms from cluster start. Timeout at 2025-06-26 05:13:04.057604923 +0000 UTC m=+12.206715201, 107.71168ms after stability
2025-06-26T05:13:04.165Z [INFO]  server-84d28f70-2755-a114-800b-d7bc710dbed8: aborting pipeline replication: peer="{Voter server-6f1067d3-a104-5938-c6e6-4e32b0b5e1fc 6f1067d3-a104-5938-c6e6-4e32b0b5e1fc}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.70s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2025-06-26T05:13:04.166Z [INFO]  server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 Address:b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5} {Suffrage:Voter ID:server-e3085a45-540d-bfcb-0543-26993d7b1fbf Address:e3085a45-540d-bfcb-0543-26993d7b1fbf} {Suffrage:Voter ID:server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae Address:01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae} {Suffrage:Voter ID:server-5314abae-63dd-94e5-9cca-41835758fd49 Address:5314abae-63dd-94e5-9cca-41835758fd49}]"
2025-06-26T05:13:04.166Z [INFO]  server-e3085a45-540d-bfcb-0543-26993d7b1fbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 Address:b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5} {Suffrage:Voter ID:server-e3085a45-540d-bfcb-0543-26993d7b1fbf Address:e3085a45-540d-bfcb-0543-26993d7b1fbf} {Suffrage:Voter ID:server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae Address:01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae} {Suffrage:Voter ID:server-5314abae-63dd-94e5-9cca-41835758fd49 Address:5314abae-63dd-94e5-9cca-41835758fd49}]"
2025-06-26T05:13:04.166Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 Address:b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5} {Suffrage:Voter ID:server-e3085a45-540d-bfcb-0543-26993d7b1fbf Address:e3085a45-540d-bfcb-0543-26993d7b1fbf} {Suffrage:Voter ID:server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae Address:01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae} {Suffrage:Voter ID:server-5314abae-63dd-94e5-9cca-41835758fd49 Address:5314abae-63dd-94e5-9cca-41835758fd49}]"
2025-06-26T05:13:04.166Z [INFO]  server-5314abae-63dd-94e5-9cca-41835758fd49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 Address:b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5} {Suffrage:Voter ID:server-e3085a45-540d-bfcb-0543-26993d7b1fbf Address:e3085a45-540d-bfcb-0543-26993d7b1fbf} {Suffrage:Voter ID:server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae Address:01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae} {Suffrage:Voter ID:server-5314abae-63dd-94e5-9cca-41835758fd49 Address:5314abae-63dd-94e5-9cca-41835758fd49}]"
2025-06-26T05:13:04.166Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:04.166Z [INFO]  server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5: entering follower state: follower="Node at b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 [Follower]" leader-address= leader-id=
2025-06-26T05:13:04.166Z [INFO]  server-e3085a45-540d-bfcb-0543-26993d7b1fbf: entering follower state: follower="Node at e3085a45-540d-bfcb-0543-26993d7b1fbf [Follower]" leader-address= leader-id=
2025-06-26T05:13:04.166Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: entering follower state: follower="Node at 01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae [Follower]" leader-address= leader-id=
2025-06-26T05:13:04.166Z [INFO]  server-5314abae-63dd-94e5-9cca-41835758fd49: entering follower state: follower="Node at 5314abae-63dd-94e5-9cca-41835758fd49 [Follower]" leader-address= leader-id=
2025-06-26T05:13:04.225Z [WARN]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:04.225Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: entering candidate state: node="Node at 01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae [Candidate]" term=2
2025-06-26T05:13:04.225Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: election won: term=2 tally=3
2025-06-26T05:13:04.225Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: entering leader state: leader="Node at 01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae [Leader]"
2025-06-26T05:13:04.225Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: added peer, starting replication: peer=server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5
2025-06-26T05:13:04.225Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: added peer, starting replication: peer=server-e3085a45-540d-bfcb-0543-26993d7b1fbf
2025-06-26T05:13:04.225Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: added peer, starting replication: peer=server-5314abae-63dd-94e5-9cca-41835758fd49
2025-06-26T05:13:04.225Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: pipelining replication: peer="{Voter server-5314abae-63dd-94e5-9cca-41835758fd49 5314abae-63dd-94e5-9cca-41835758fd49}"
2025-06-26T05:13:04.225Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: pipelining replication: peer="{Voter server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5}"
2025-06-26T05:13:04.226Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: pipelining replication: peer="{Voter server-e3085a45-540d-bfcb-0543-26993d7b1fbf e3085a45-540d-bfcb-0543-26993d7b1fbf}"
2025-06-26T05:13:04.333Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:04.225966163 +0000 UTC m=+12.375076401 (1 nodes), 59.62228ms from start of poll, 59.87372ms from cluster start. Timeout at 2025-06-26 05:13:04.333607163 +0000 UTC m=+12.482717441, 107.64104ms after stability
2025-06-26T05:13:04.441Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:04.553Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:04.441681443 +0000 UTC m=+12.590791681 (3 nodes), 2.44µs from start of poll, 275.589ms from cluster start. Timeout at 2025-06-26 05:13:04.553602963 +0000 UTC m=+12.702713241, 111.92156ms after stability
2025-06-26T05:13:04.553Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: updating configuration: command=RemoveServer server-id=server-5314abae-63dd-94e5-9cca-41835758fd49 server-addr= servers="[{Suffrage:Voter ID:server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 Address:b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5} {Suffrage:Voter ID:server-e3085a45-540d-bfcb-0543-26993d7b1fbf Address:e3085a45-540d-bfcb-0543-26993d7b1fbf} {Suffrage:Voter ID:server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae Address:01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae}]"
2025-06-26T05:13:04.553Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: removed peer, stopping replication: peer=server-5314abae-63dd-94e5-9cca-41835758fd49 last-index=3
2025-06-26T05:13:04.553Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: aborting pipeline replication: peer="{Voter server-5314abae-63dd-94e5-9cca-41835758fd49 5314abae-63dd-94e5-9cca-41835758fd49}"
2025-06-26T05:13:04.553Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to heartbeat to: peer=e3085a45-540d-bfcb-0543-26993d7b1fbf backoff time=10ms error="failed to connect to peer: e3085a45-540d-bfcb-0543-26993d7b1fbf"
2025-06-26T05:13:04.553Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to pipeline appendEntries: peer="{Voter server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5}" error="append pipeline closed"
2025-06-26T05:13:04.553Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: aborting pipeline replication: peer="{Voter server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5}"
2025-06-26T05:13:04.553Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to pipeline appendEntries: peer="{Voter server-e3085a45-540d-bfcb-0543-26993d7b1fbf e3085a45-540d-bfcb-0543-26993d7b1fbf}" error="append pipeline closed"
2025-06-26T05:13:04.553Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: aborting pipeline replication: peer="{Voter server-e3085a45-540d-bfcb-0543-26993d7b1fbf e3085a45-540d-bfcb-0543-26993d7b1fbf}"
2025-06-26T05:13:04.561Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to appendEntries to: peer="{Voter server-e3085a45-540d-bfcb-0543-26993d7b1fbf e3085a45-540d-bfcb-0543-26993d7b1fbf}" error="failed to connect to peer: e3085a45-540d-bfcb-0543-26993d7b1fbf"
2025-06-26T05:13:04.565Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to appendEntries to: peer="{Voter server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5}" error="failed to connect to peer: b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5"
2025-06-26T05:13:04.565Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to heartbeat to: peer=b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 backoff time=10ms error="failed to connect to peer: b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5"
2025-06-26T05:13:04.577Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to heartbeat to: peer=e3085a45-540d-bfcb-0543-26993d7b1fbf backoff time=10ms error="failed to connect to peer: e3085a45-540d-bfcb-0543-26993d7b1fbf"
2025-06-26T05:13:04.581Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to appendEntries to: peer="{Voter server-e3085a45-540d-bfcb-0543-26993d7b1fbf e3085a45-540d-bfcb-0543-26993d7b1fbf}" error="failed to connect to peer: e3085a45-540d-bfcb-0543-26993d7b1fbf"
2025-06-26T05:13:04.585Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to appendEntries to: peer="{Voter server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5}" error="failed to connect to peer: b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5"
2025-06-26T05:13:04.589Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to heartbeat to: peer=b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 backoff time=10ms error="failed to connect to peer: b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5"
2025-06-26T05:13:04.597Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to heartbeat to: peer=e3085a45-540d-bfcb-0543-26993d7b1fbf backoff time=10ms error="failed to connect to peer: e3085a45-540d-bfcb-0543-26993d7b1fbf"
2025-06-26T05:13:04.597Z [WARN]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to contact: server-id=server-e3085a45-540d-bfcb-0543-26993d7b1fbf time=55.82088ms
2025-06-26T05:13:04.605Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to appendEntries to: peer="{Voter server-e3085a45-540d-bfcb-0543-26993d7b1fbf e3085a45-540d-bfcb-0543-26993d7b1fbf}" error="failed to connect to peer: e3085a45-540d-bfcb-0543-26993d7b1fbf"
2025-06-26T05:13:04.609Z [ERROR] server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to appendEntries to: peer="{Voter server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5}" error="failed to connect to peer: b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5"
2025-06-26T05:13:04.609Z [WARN]  server-e3085a45-540d-bfcb-0543-26993d7b1fbf: heartbeat timeout reached, starting election: last-leader-addr=01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae last-leader-id=server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae
2025-06-26T05:13:04.609Z [INFO]  server-e3085a45-540d-bfcb-0543-26993d7b1fbf: entering candidate state: node="Node at e3085a45-540d-bfcb-0543-26993d7b1fbf [Candidate]" term=3
2025-06-26T05:13:04.609Z [ERROR] server-e3085a45-540d-bfcb-0543-26993d7b1fbf: failed to make requestVote RPC: target="{Voter server-5314abae-63dd-94e5-9cca-41835758fd49 5314abae-63dd-94e5-9cca-41835758fd49}" error="failed to connect to peer: 5314abae-63dd-94e5-9cca-41835758fd49" term=3
2025-06-26T05:13:04.609Z [WARN]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to contact: server-id=server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 time=56.10504ms
2025-06-26T05:13:04.609Z [WARN]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to contact: server-id=server-e3085a45-540d-bfcb-0543-26993d7b1fbf time=68.01004ms
2025-06-26T05:13:04.609Z [WARN]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: failed to contact quorum of nodes, stepping down
2025-06-26T05:13:04.609Z [INFO]  server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae: entering follower state: follower="Node at 01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae [Follower]" leader-address= leader-id=
2025-06-26T05:13:04.609Z [ERROR] server-e3085a45-540d-bfcb-0543-26993d7b1fbf: failed to make requestVote RPC: target="{Voter server-01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae 01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae}" error="failed to connect to peer: 01df0f35-b8ef-46f1-66f4-5b9b5bd4a5ae" term=3
2025-06-26T05:13:05.113Z [ERROR] server-e3085a45-540d-bfcb-0543-26993d7b1fbf: failed to make requestVote RPC: target="{Voter server-b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5 b7ed4fd6-b5ee-140e-ef38-16cd1e5797e5}" error="command timed out" term=3
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.95s)
=== RUN   TestRaft_AddKnownPeer
2025-06-26T05:13:05.114Z [INFO]  server-b6583126-6f8e-feb2-210d-27a5d338bc86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6583126-6f8e-feb2-210d-27a5d338bc86 Address:b6583126-6f8e-feb2-210d-27a5d338bc86} {Suffrage:Voter ID:server-c86ecac8-9710-accc-0171-6fe19cf7a425 Address:c86ecac8-9710-accc-0171-6fe19cf7a425} {Suffrage:Voter ID:server-3e790a5d-337e-ea93-c8a2-50a3efc7fdad Address:3e790a5d-337e-ea93-c8a2-50a3efc7fdad}]"
2025-06-26T05:13:05.114Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6583126-6f8e-feb2-210d-27a5d338bc86 Address:b6583126-6f8e-feb2-210d-27a5d338bc86} {Suffrage:Voter ID:server-c86ecac8-9710-accc-0171-6fe19cf7a425 Address:c86ecac8-9710-accc-0171-6fe19cf7a425} {Suffrage:Voter ID:server-3e790a5d-337e-ea93-c8a2-50a3efc7fdad Address:3e790a5d-337e-ea93-c8a2-50a3efc7fdad}]"
2025-06-26T05:13:05.114Z [INFO]  server-3e790a5d-337e-ea93-c8a2-50a3efc7fdad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6583126-6f8e-feb2-210d-27a5d338bc86 Address:b6583126-6f8e-feb2-210d-27a5d338bc86} {Suffrage:Voter ID:server-c86ecac8-9710-accc-0171-6fe19cf7a425 Address:c86ecac8-9710-accc-0171-6fe19cf7a425} {Suffrage:Voter ID:server-3e790a5d-337e-ea93-c8a2-50a3efc7fdad Address:3e790a5d-337e-ea93-c8a2-50a3efc7fdad}]"
2025-06-26T05:13:05.114Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:05.114Z [INFO]  server-b6583126-6f8e-feb2-210d-27a5d338bc86: entering follower state: follower="Node at b6583126-6f8e-feb2-210d-27a5d338bc86 [Follower]" leader-address= leader-id=
2025-06-26T05:13:05.114Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: entering follower state: follower="Node at c86ecac8-9710-accc-0171-6fe19cf7a425 [Follower]" leader-address= leader-id=
2025-06-26T05:13:05.114Z [INFO]  server-3e790a5d-337e-ea93-c8a2-50a3efc7fdad: entering follower state: follower="Node at 3e790a5d-337e-ea93-c8a2-50a3efc7fdad [Follower]" leader-address= leader-id=
2025-06-26T05:13:05.201Z [WARN]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:05.201Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: entering candidate state: node="Node at c86ecac8-9710-accc-0171-6fe19cf7a425 [Candidate]" term=2
2025-06-26T05:13:05.201Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: election won: term=2 tally=2
2025-06-26T05:13:05.201Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: entering leader state: leader="Node at c86ecac8-9710-accc-0171-6fe19cf7a425 [Leader]"
2025-06-26T05:13:05.201Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: added peer, starting replication: peer=server-b6583126-6f8e-feb2-210d-27a5d338bc86
2025-06-26T05:13:05.201Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: added peer, starting replication: peer=server-3e790a5d-337e-ea93-c8a2-50a3efc7fdad
2025-06-26T05:13:05.201Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: pipelining replication: peer="{Voter server-3e790a5d-337e-ea93-c8a2-50a3efc7fdad 3e790a5d-337e-ea93-c8a2-50a3efc7fdad}"
2025-06-26T05:13:05.201Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: pipelining replication: peer="{Voter server-b6583126-6f8e-feb2-210d-27a5d338bc86 b6583126-6f8e-feb2-210d-27a5d338bc86}"
2025-06-26T05:13:05.309Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:05.201983123 +0000 UTC m=+13.351093401 (1 nodes), 87.662ms from start of poll, 87.85932ms from cluster start. Timeout at 2025-06-26 05:13:05.309714283 +0000 UTC m=+13.458824561, 107.73116ms after stability
2025-06-26T05:13:05.309Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:05.417Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:05.309764563 +0000 UTC m=+13.458874841 (2 nodes), 680ns from start of poll, 195.64076ms from cluster start. Timeout at 2025-06-26 05:13:05.417601323 +0000 UTC m=+13.566711561, 107.83672ms after stability
2025-06-26T05:13:05.417Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: updating configuration: command=AddVoter server-id=server-b6583126-6f8e-feb2-210d-27a5d338bc86 server-addr=b6583126-6f8e-feb2-210d-27a5d338bc86 servers="[{Suffrage:Voter ID:server-b6583126-6f8e-feb2-210d-27a5d338bc86 Address:b6583126-6f8e-feb2-210d-27a5d338bc86} {Suffrage:Voter ID:server-c86ecac8-9710-accc-0171-6fe19cf7a425 Address:c86ecac8-9710-accc-0171-6fe19cf7a425} {Suffrage:Voter ID:server-3e790a5d-337e-ea93-c8a2-50a3efc7fdad Address:3e790a5d-337e-ea93-c8a2-50a3efc7fdad}]"
2025-06-26T05:13:05.417Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: aborting pipeline replication: peer="{Voter server-3e790a5d-337e-ea93-c8a2-50a3efc7fdad 3e790a5d-337e-ea93-c8a2-50a3efc7fdad}"
2025-06-26T05:13:05.417Z [INFO]  server-c86ecac8-9710-accc-0171-6fe19cf7a425: aborting pipeline replication: peer="{Voter server-b6583126-6f8e-feb2-210d-27a5d338bc86 b6583126-6f8e-feb2-210d-27a5d338bc86}"
--- PASS: TestRaft_AddKnownPeer (0.30s)
=== RUN   TestRaft_RemoveUnknownPeer
2025-06-26T05:13:05.418Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8 Address:b76568a5-a13b-f29d-7ca6-f134a9df3ce8} {Suffrage:Voter ID:server-5a1d32a6-8ecf-d1e8-f0bf-23d16300de02 Address:5a1d32a6-8ecf-d1e8-f0bf-23d16300de02} {Suffrage:Voter ID:server-f5b18129-b895-1278-7224-654a96bf8158 Address:f5b18129-b895-1278-7224-654a96bf8158}]"
2025-06-26T05:13:05.418Z [INFO]  server-5a1d32a6-8ecf-d1e8-f0bf-23d16300de02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8 Address:b76568a5-a13b-f29d-7ca6-f134a9df3ce8} {Suffrage:Voter ID:server-5a1d32a6-8ecf-d1e8-f0bf-23d16300de02 Address:5a1d32a6-8ecf-d1e8-f0bf-23d16300de02} {Suffrage:Voter ID:server-f5b18129-b895-1278-7224-654a96bf8158 Address:f5b18129-b895-1278-7224-654a96bf8158}]"
2025-06-26T05:13:05.418Z [INFO]  server-f5b18129-b895-1278-7224-654a96bf8158: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8 Address:b76568a5-a13b-f29d-7ca6-f134a9df3ce8} {Suffrage:Voter ID:server-5a1d32a6-8ecf-d1e8-f0bf-23d16300de02 Address:5a1d32a6-8ecf-d1e8-f0bf-23d16300de02} {Suffrage:Voter ID:server-f5b18129-b895-1278-7224-654a96bf8158 Address:f5b18129-b895-1278-7224-654a96bf8158}]"
2025-06-26T05:13:05.418Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:05.418Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: entering follower state: follower="Node at b76568a5-a13b-f29d-7ca6-f134a9df3ce8 [Follower]" leader-address= leader-id=
2025-06-26T05:13:05.418Z [INFO]  server-5a1d32a6-8ecf-d1e8-f0bf-23d16300de02: entering follower state: follower="Node at 5a1d32a6-8ecf-d1e8-f0bf-23d16300de02 [Follower]" leader-address= leader-id=
2025-06-26T05:13:05.418Z [INFO]  server-f5b18129-b895-1278-7224-654a96bf8158: entering follower state: follower="Node at f5b18129-b895-1278-7224-654a96bf8158 [Follower]" leader-address= leader-id=
2025-06-26T05:13:05.481Z [WARN]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:05.481Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: entering candidate state: node="Node at b76568a5-a13b-f29d-7ca6-f134a9df3ce8 [Candidate]" term=2
2025-06-26T05:13:05.481Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: election won: term=2 tally=2
2025-06-26T05:13:05.481Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: entering leader state: leader="Node at b76568a5-a13b-f29d-7ca6-f134a9df3ce8 [Leader]"
2025-06-26T05:13:05.481Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: added peer, starting replication: peer=server-5a1d32a6-8ecf-d1e8-f0bf-23d16300de02
2025-06-26T05:13:05.481Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: added peer, starting replication: peer=server-f5b18129-b895-1278-7224-654a96bf8158
2025-06-26T05:13:05.481Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: pipelining replication: peer="{Voter server-f5b18129-b895-1278-7224-654a96bf8158 f5b18129-b895-1278-7224-654a96bf8158}"
2025-06-26T05:13:05.481Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: pipelining replication: peer="{Voter server-5a1d32a6-8ecf-d1e8-f0bf-23d16300de02 5a1d32a6-8ecf-d1e8-f0bf-23d16300de02}"
2025-06-26T05:13:05.589Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:05.481969563 +0000 UTC m=+13.631079841 (1 nodes), 63.42124ms from start of poll, 63.59596ms from cluster start. Timeout at 2025-06-26 05:13:05.589610803 +0000 UTC m=+13.738721081, 107.64124ms after stability
2025-06-26T05:13:05.589Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: updating configuration: command=RemoveServer server-id=02df6dbe-3f02-eb94-50a1-b90251e70da5 server-addr= servers="[{Suffrage:Voter ID:server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8 Address:b76568a5-a13b-f29d-7ca6-f134a9df3ce8} {Suffrage:Voter ID:server-5a1d32a6-8ecf-d1e8-f0bf-23d16300de02 Address:5a1d32a6-8ecf-d1e8-f0bf-23d16300de02} {Suffrage:Voter ID:server-f5b18129-b895-1278-7224-654a96bf8158 Address:f5b18129-b895-1278-7224-654a96bf8158}]"
2025-06-26T05:13:05.589Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: aborting pipeline replication: peer="{Voter server-f5b18129-b895-1278-7224-654a96bf8158 f5b18129-b895-1278-7224-654a96bf8158}"
2025-06-26T05:13:05.589Z [INFO]  server-b76568a5-a13b-f29d-7ca6-f134a9df3ce8: aborting pipeline replication: peer="{Voter server-5a1d32a6-8ecf-d1e8-f0bf-23d16300de02 5a1d32a6-8ecf-d1e8-f0bf-23d16300de02}"
--- PASS: TestRaft_RemoveUnknownPeer (0.17s)
=== RUN   TestRaft_SnapshotRestore
2025-06-26T05:13:05.590Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13e5eb1d-1e28-2fdf-53db-d890da482656 Address:13e5eb1d-1e28-2fdf-53db-d890da482656}]"
2025-06-26T05:13:05.590Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:05.590Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: entering follower state: follower="Node at 13e5eb1d-1e28-2fdf-53db-d890da482656 [Follower]" leader-address= leader-id=
2025-06-26T05:13:05.657Z [WARN]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:05.657Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: entering candidate state: node="Node at 13e5eb1d-1e28-2fdf-53db-d890da482656 [Candidate]" term=2
2025-06-26T05:13:05.657Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: election won: term=2 tally=1
2025-06-26T05:13:05.657Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: entering leader state: leader="Node at 13e5eb1d-1e28-2fdf-53db-d890da482656 [Leader]"
2025-06-26T05:13:05.763Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:05.657737483 +0000 UTC m=+13.806847761 (1 nodes), 67.44284ms from start of poll, 67.52328ms from cluster start. Timeout at 2025-06-26 05:13:05.763187723 +0000 UTC m=+13.912298001, 105.45024ms after stability
2025-06-26T05:13:05.764Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: starting snapshot up to: index=102
2025-06-26T05:13:05.764Z [INFO]  creating new snapshot: path=/tmp/raft490792680/snapshots/2-102-1750914785764.tmp
2025-06-26T05:13:05.764Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: compacting logs: from=1 to=92
2025-06-26T05:13:05.764Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: snapshot complete up to: index=102
2025-06-26T05:13:05.764Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: starting restore from snapshot: id=2-102-1750914785764 last-index=102 last-term=2 size-in-bytes=693
2025-06-26T05:13:05.781Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: snapshot restore progress: id=2-102-1750914785764 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-06-26T05:13:05.781Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: restored from snapshot: id=2-102-1750914785764 last-index=102 last-term=2 size-in-bytes=693
2025-06-26T05:13:05.781Z [INFO]  server-13e5eb1d-1e28-2fdf-53db-d890da482656: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13e5eb1d-1e28-2fdf-53db-d890da482656 Address:13e5eb1d-1e28-2fdf-53db-d890da482656}]"
--- PASS: TestRaft_SnapshotRestore (0.19s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2025-06-26T05:13:05.782Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a Address:62fe95d7-a019-9f9d-7425-8b1c9ce7f63a}]"
2025-06-26T05:13:05.782Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:05.782Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: entering follower state: follower="Node at 62fe95d7-a019-9f9d-7425-8b1c9ce7f63a [Follower]" leader-address= leader-id=
2025-06-26T05:13:05.841Z [WARN]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:05.841Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: entering candidate state: node="Node at 62fe95d7-a019-9f9d-7425-8b1c9ce7f63a [Candidate]" term=2
2025-06-26T05:13:05.841Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: election won: term=2 tally=1
2025-06-26T05:13:05.841Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: entering leader state: leader="Node at 62fe95d7-a019-9f9d-7425-8b1c9ce7f63a [Leader]"
2025-06-26T05:13:05.949Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:05.841775883 +0000 UTC m=+13.990886161 (1 nodes), 59.65028ms from start of poll, 59.77564ms from cluster start. Timeout at 2025-06-26 05:13:05.949578003 +0000 UTC m=+14.098688281, 107.80212ms after stability
2025-06-26T05:13:05.950Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: starting snapshot up to: index=102
2025-06-26T05:13:05.950Z [INFO]  creating new snapshot: path=/tmp/raft1487080084/snapshots/2-102-1750914785950.tmp
2025-06-26T05:13:05.951Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: compacting logs: from=1 to=92
2025-06-26T05:13:05.951Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: snapshot complete up to: index=102
2025-06-26T05:13:05.951Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: starting restore from snapshot: id=2-102-1750914785950 last-index=102 last-term=2 size-in-bytes=693
2025-06-26T05:13:05.951Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: snapshot restore progress: id=2-102-1750914785950 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-06-26T05:13:05.951Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: restored from snapshot: id=2-102-1750914785950 last-index=102 last-term=2 size-in-bytes=693
2025-06-26T05:13:05.951Z [INFO]  server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62fe95d7-a019-9f9d-7425-8b1c9ce7f63a Address:62fe95d7-a019-9f9d-7425-8b1c9ce7f63a}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.17s)
=== RUN   TestRaft_SnapshotRestore_Progress
2025-06-26T05:13:05.951Z [INFO]  server-84bd894a-1a64-f777-61f5-1d5f6e64a5dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84bd894a-1a64-f777-61f5-1d5f6e64a5dc Address:84bd894a-1a64-f777-61f5-1d5f6e64a5dc}]"
2025-06-26T05:13:05.951Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:05.951Z [INFO]  server-84bd894a-1a64-f777-61f5-1d5f6e64a5dc: entering follower state: follower="Node at 84bd894a-1a64-f777-61f5-1d5f6e64a5dc [Follower]" leader-address= leader-id=
2025-06-26T05:13:06.005Z [WARN]  server-84bd894a-1a64-f777-61f5-1d5f6e64a5dc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:06.005Z [INFO]  server-84bd894a-1a64-f777-61f5-1d5f6e64a5dc: entering candidate state: node="Node at 84bd894a-1a64-f777-61f5-1d5f6e64a5dc [Candidate]" term=2
2025-06-26T05:13:06.005Z [INFO]  server-84bd894a-1a64-f777-61f5-1d5f6e64a5dc: election won: term=2 tally=1
2025-06-26T05:13:06.005Z [INFO]  server-84bd894a-1a64-f777-61f5-1d5f6e64a5dc: entering leader state: leader="Node at 84bd894a-1a64-f777-61f5-1d5f6e64a5dc [Leader]"
2025-06-26T05:13:06.113Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:06.005772483 +0000 UTC m=+14.154882761 (1 nodes), 54.20156ms from start of poll, 54.27176ms from cluster start. Timeout at 2025-06-26 05:13:06.113600123 +0000 UTC m=+14.262710401, 107.82764ms after stability
2025-06-26T05:13:06.114Z [INFO]  server-84bd894a-1a64-f777-61f5-1d5f6e64a5dc: starting snapshot up to: index=102
2025-06-26T05:13:06.114Z [INFO]  creating new snapshot: path=/tmp/raft1954879502/snapshots/2-102-1750914786114.tmp
2025-06-26T05:13:06.115Z [INFO]  server-84bd894a-1a64-f777-61f5-1d5f6e64a5dc: compacting logs: from=1 to=92
2025-06-26T05:13:06.115Z [INFO]  server-84bd894a-1a64-f777-61f5-1d5f6e64a5dc: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.16s)
=== RUN   TestRaft_NoRestoreOnStart
2025-06-26T05:13:06.115Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-420233f6-fe35-d6eb-c455-741af7fd6c82 Address:420233f6-fe35-d6eb-c455-741af7fd6c82}]"
2025-06-26T05:13:06.115Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:06.115Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: entering follower state: follower="Node at 420233f6-fe35-d6eb-c455-741af7fd6c82 [Follower]" leader-address= leader-id=
2025-06-26T05:13:06.177Z [WARN]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:06.177Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: entering candidate state: node="Node at 420233f6-fe35-d6eb-c455-741af7fd6c82 [Candidate]" term=2
2025-06-26T05:13:06.177Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: election won: term=2 tally=1
2025-06-26T05:13:06.177Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: entering leader state: leader="Node at 420233f6-fe35-d6eb-c455-741af7fd6c82 [Leader]"
2025-06-26T05:13:06.285Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:06.177769923 +0000 UTC m=+14.326880201 (1 nodes), 62.09496ms from start of poll, 62.17544ms from cluster start. Timeout at 2025-06-26 05:13:06.285579723 +0000 UTC m=+14.434690241, 107.81004ms after stability
2025-06-26T05:13:06.286Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: starting snapshot up to: index=102
2025-06-26T05:13:06.286Z [INFO]  creating new snapshot: path=/tmp/raft2395118970/snapshots/2-102-1750914786286.tmp
2025-06-26T05:13:06.287Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: compacting logs: from=1 to=92
2025-06-26T05:13:06.287Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: snapshot complete up to: index=102
2025-06-26T05:13:06.287Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-420233f6-fe35-d6eb-c455-741af7fd6c82 Address:420233f6-fe35-d6eb-c455-741af7fd6c82}]"
--- PASS: TestRaft_NoRestoreOnStart (0.17s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2025-06-26T05:13:06.287Z [INFO]  server-fae6a35c-6ff7-0dbf-ddb0-f640739fd947: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fae6a35c-6ff7-0dbf-ddb0-f640739fd947 Address:fae6a35c-6ff7-0dbf-ddb0-f640739fd947}]"
2025-06-26T05:13:06.287Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:06.287Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: entering follower state: follower="Node at 420233f6-fe35-d6eb-c455-741af7fd6c82 [Follower]" leader-address= leader-id=
2025-06-26T05:13:06.287Z [INFO]  server-fae6a35c-6ff7-0dbf-ddb0-f640739fd947: entering follower state: follower="Node at fae6a35c-6ff7-0dbf-ddb0-f640739fd947 [Follower]" leader-address= leader-id=
2025-06-26T05:13:06.345Z [WARN]  server-fae6a35c-6ff7-0dbf-ddb0-f640739fd947: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:06.345Z [INFO]  server-fae6a35c-6ff7-0dbf-ddb0-f640739fd947: entering candidate state: node="Node at fae6a35c-6ff7-0dbf-ddb0-f640739fd947 [Candidate]" term=2
2025-06-26T05:13:06.345Z [INFO]  server-fae6a35c-6ff7-0dbf-ddb0-f640739fd947: election won: term=2 tally=1
2025-06-26T05:13:06.345Z [INFO]  server-fae6a35c-6ff7-0dbf-ddb0-f640739fd947: entering leader state: leader="Node at fae6a35c-6ff7-0dbf-ddb0-f640739fd947 [Leader]"
2025-06-26T05:13:06.385Z [WARN]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:06.385Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: entering candidate state: node="Node at 420233f6-fe35-d6eb-c455-741af7fd6c82 [Candidate]" term=3
2025-06-26T05:13:06.385Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: election won: term=3 tally=1
2025-06-26T05:13:06.385Z [INFO]  server-420233f6-fe35-d6eb-c455-741af7fd6c82: entering leader state: leader="Node at 420233f6-fe35-d6eb-c455-741af7fd6c82 [Leader]"
2025-06-26T05:13:06.453Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:06.345795083 +0000 UTC m=+14.494905361 (1 nodes), 58.4506ms from start of poll, 58.52716ms from cluster start. Timeout at 2025-06-26 05:13:06.453570523 +0000 UTC m=+14.602680801, 107.77544ms after stability
2025-06-26T05:13:06.469Z [INFO]  server-fae6a35c-6ff7-0dbf-ddb0-f640739fd947: starting snapshot up to: index=102
2025-06-26T05:13:06.469Z [INFO]  creating new snapshot: path=/tmp/raft2288168946/snapshots/2-102-1750914786469.tmp
2025-06-26T05:13:06.469Z [INFO]  server-fae6a35c-6ff7-0dbf-ddb0-f640739fd947: compacting logs: from=1 to=92
2025-06-26T05:13:06.470Z [INFO]  server-fae6a35c-6ff7-0dbf-ddb0-f640739fd947: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.27s)
=== RUN   TestRaft_UserSnapshot
2025-06-26T05:13:06.562Z [INFO]  server-5cac087a-044c-a3a9-8baf-f118bee67658: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5cac087a-044c-a3a9-8baf-f118bee67658 Address:5cac087a-044c-a3a9-8baf-f118bee67658}]"
2025-06-26T05:13:06.562Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:06.562Z [INFO]  server-5cac087a-044c-a3a9-8baf-f118bee67658: entering follower state: follower="Node at 5cac087a-044c-a3a9-8baf-f118bee67658 [Follower]" leader-address= leader-id=
2025-06-26T05:13:06.641Z [WARN]  server-5cac087a-044c-a3a9-8baf-f118bee67658: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:06.641Z [INFO]  server-5cac087a-044c-a3a9-8baf-f118bee67658: entering candidate state: node="Node at 5cac087a-044c-a3a9-8baf-f118bee67658 [Candidate]" term=2
2025-06-26T05:13:06.641Z [INFO]  server-5cac087a-044c-a3a9-8baf-f118bee67658: election won: term=2 tally=1
2025-06-26T05:13:06.641Z [INFO]  server-5cac087a-044c-a3a9-8baf-f118bee67658: entering leader state: leader="Node at 5cac087a-044c-a3a9-8baf-f118bee67658 [Leader]"
2025-06-26T05:13:06.749Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:06.641741963 +0000 UTC m=+14.790852241 (1 nodes), 79.706ms from start of poll, 79.80784ms from cluster start. Timeout at 2025-06-26 05:13:06.749599563 +0000 UTC m=+14.898709841, 107.8576ms after stability
2025-06-26T05:13:06.765Z [ERROR] server-5cac087a-044c-a3a9-8baf-f118bee67658: failed to take snapshot: error="nothing new to snapshot"
2025-06-26T05:13:06.766Z [INFO]  server-5cac087a-044c-a3a9-8baf-f118bee67658: starting snapshot up to: index=12
2025-06-26T05:13:06.766Z [INFO]  creating new snapshot: path=/tmp/raft2238126330/snapshots/2-12-1750914786766.tmp
2025-06-26T05:13:06.766Z [INFO]  server-5cac087a-044c-a3a9-8baf-f118bee67658: compacting logs: from=1 to=2
2025-06-26T05:13:06.766Z [INFO]  server-5cac087a-044c-a3a9-8baf-f118bee67658: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.20s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2025-06-26T05:13:06.766Z [INFO]  server-317853c2-4e22-8096-56af-6e5b6339a79f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-317853c2-4e22-8096-56af-6e5b6339a79f Address:317853c2-4e22-8096-56af-6e5b6339a79f} {Suffrage:Voter ID:server-53a5bdd8-e787-8798-3106-931ce3f90d29 Address:53a5bdd8-e787-8798-3106-931ce3f90d29} {Suffrage:Voter ID:server-b012d426-8e28-f805-4049-7143afeab2fc Address:b012d426-8e28-f805-4049-7143afeab2fc}]"
2025-06-26T05:13:06.766Z [INFO]  server-53a5bdd8-e787-8798-3106-931ce3f90d29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-317853c2-4e22-8096-56af-6e5b6339a79f Address:317853c2-4e22-8096-56af-6e5b6339a79f} {Suffrage:Voter ID:server-53a5bdd8-e787-8798-3106-931ce3f90d29 Address:53a5bdd8-e787-8798-3106-931ce3f90d29} {Suffrage:Voter ID:server-b012d426-8e28-f805-4049-7143afeab2fc Address:b012d426-8e28-f805-4049-7143afeab2fc}]"
2025-06-26T05:13:06.766Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-317853c2-4e22-8096-56af-6e5b6339a79f Address:317853c2-4e22-8096-56af-6e5b6339a79f} {Suffrage:Voter ID:server-53a5bdd8-e787-8798-3106-931ce3f90d29 Address:53a5bdd8-e787-8798-3106-931ce3f90d29} {Suffrage:Voter ID:server-b012d426-8e28-f805-4049-7143afeab2fc Address:b012d426-8e28-f805-4049-7143afeab2fc}]"
2025-06-26T05:13:06.766Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:06.766Z [INFO]  server-317853c2-4e22-8096-56af-6e5b6339a79f: entering follower state: follower="Node at 317853c2-4e22-8096-56af-6e5b6339a79f [Follower]" leader-address= leader-id=
2025-06-26T05:13:06.766Z [INFO]  server-53a5bdd8-e787-8798-3106-931ce3f90d29: entering follower state: follower="Node at 53a5bdd8-e787-8798-3106-931ce3f90d29 [Follower]" leader-address= leader-id=
2025-06-26T05:13:06.766Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: entering follower state: follower="Node at b012d426-8e28-f805-4049-7143afeab2fc [Follower]" leader-address= leader-id=
2025-06-26T05:13:07.309Z [WARN]  server-b012d426-8e28-f805-4049-7143afeab2fc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:07.309Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: entering candidate state: node="Node at b012d426-8e28-f805-4049-7143afeab2fc [Candidate]" term=2
2025-06-26T05:13:07.309Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: election won: term=2 tally=2
2025-06-26T05:13:07.309Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: entering leader state: leader="Node at b012d426-8e28-f805-4049-7143afeab2fc [Leader]"
2025-06-26T05:13:07.309Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: added peer, starting replication: peer=server-317853c2-4e22-8096-56af-6e5b6339a79f
2025-06-26T05:13:07.309Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: added peer, starting replication: peer=server-53a5bdd8-e787-8798-3106-931ce3f90d29
2025-06-26T05:13:07.309Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: pipelining replication: peer="{Voter server-53a5bdd8-e787-8798-3106-931ce3f90d29 53a5bdd8-e787-8798-3106-931ce3f90d29}"
2025-06-26T05:13:07.309Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: pipelining replication: peer="{Voter server-317853c2-4e22-8096-56af-6e5b6339a79f 317853c2-4e22-8096-56af-6e5b6339a79f}"
2025-06-26T05:13:08.317Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:07.309976283 +0000 UTC m=+15.459086561 (1 nodes), 543.23452ms from start of poll, 543.42272ms from cluster start. Timeout at 2025-06-26 05:13:08.317613963 +0000 UTC m=+16.466724241, 1.00763768s after stability
2025-06-26T05:13:08.318Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: starting snapshot up to: index=12
2025-06-26T05:13:08.318Z [INFO]  creating new snapshot: path=/tmp/raft2515521191/snapshots/2-12-1750914788318.tmp
2025-06-26T05:13:08.318Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: snapshot complete up to: index=12
2025-06-26T05:13:08.318Z [INFO]  creating new snapshot: path=/tmp/raft2515521191/snapshots/2-23-1750914788318.tmp
2025-06-26T05:13:08.319Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: copied to local snapshot: bytes=71
2025-06-26T05:13:08.319Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: snapshot restore progress: id=2-23-1750914788318 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:08.319Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: restored user snapshot: index=23
2025-06-26T05:13:08.319Z [ERROR] server-b012d426-8e28-f805-4049-7143afeab2fc: failed to get log: index=23 error="log not found"
2025-06-26T05:13:08.319Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: aborting pipeline replication: peer="{Voter server-53a5bdd8-e787-8798-3106-931ce3f90d29 53a5bdd8-e787-8798-3106-931ce3f90d29}"
2025-06-26T05:13:08.319Z [ERROR] server-b012d426-8e28-f805-4049-7143afeab2fc: failed to get log: index=23 error="log not found"
2025-06-26T05:13:08.319Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: aborting pipeline replication: peer="{Voter server-317853c2-4e22-8096-56af-6e5b6339a79f 317853c2-4e22-8096-56af-6e5b6339a79f}"
2025-06-26T05:13:08.329Z [ERROR] server-b012d426-8e28-f805-4049-7143afeab2fc: failed to get log: index=23 error="log not found"
2025-06-26T05:13:08.330Z [ERROR] server-b012d426-8e28-f805-4049-7143afeab2fc: failed to get log: index=23 error="log not found"
2025-06-26T05:13:08.331Z [INFO]  creating new snapshot: path=/tmp/raft491674500/snapshots/2-23-1750914788331.tmp
2025-06-26T05:13:08.331Z [INFO]  server-317853c2-4e22-8096-56af-6e5b6339a79f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:08.331Z [INFO]  server-317853c2-4e22-8096-56af-6e5b6339a79f: copied to local snapshot: bytes=71
2025-06-26T05:13:08.331Z [INFO]  server-317853c2-4e22-8096-56af-6e5b6339a79f: snapshot restore progress: id=2-23-1750914788331 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:08.331Z [INFO]  server-317853c2-4e22-8096-56af-6e5b6339a79f: Installed remote snapshot
2025-06-26T05:13:08.331Z [WARN]  server-317853c2-4e22-8096-56af-6e5b6339a79f: skipping application of old log: index=22
2025-06-26T05:13:08.331Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: pipelining replication: peer="{Voter server-317853c2-4e22-8096-56af-6e5b6339a79f 317853c2-4e22-8096-56af-6e5b6339a79f}"
2025-06-26T05:13:08.331Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:08.337Z [INFO]  creating new snapshot: path=/tmp/raft2201594403/snapshots/2-23-1750914788337.tmp
2025-06-26T05:13:08.338Z [INFO]  server-53a5bdd8-e787-8798-3106-931ce3f90d29: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:08.338Z [INFO]  server-53a5bdd8-e787-8798-3106-931ce3f90d29: copied to local snapshot: bytes=71
2025-06-26T05:13:08.338Z [INFO]  server-53a5bdd8-e787-8798-3106-931ce3f90d29: snapshot restore progress: id=2-23-1750914788337 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:08.338Z [INFO]  server-53a5bdd8-e787-8798-3106-931ce3f90d29: Installed remote snapshot
2025-06-26T05:13:08.338Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: pipelining replication: peer="{Voter server-53a5bdd8-e787-8798-3106-931ce3f90d29 53a5bdd8-e787-8798-3106-931ce3f90d29}"
2025-06-26T05:13:09.337Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:08.331738963 +0000 UTC m=+16.480849241 (2 nodes), 480ns from start of poll, 1.5651854s from cluster start. Timeout at 2025-06-26 05:13:09.337604083 +0000 UTC m=+17.486714401, 1.00586516s after stability
2025-06-26T05:13:09.353Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: aborting pipeline replication: peer="{Voter server-53a5bdd8-e787-8798-3106-931ce3f90d29 53a5bdd8-e787-8798-3106-931ce3f90d29}"
2025-06-26T05:13:09.353Z [INFO]  server-b012d426-8e28-f805-4049-7143afeab2fc: aborting pipeline replication: peer="{Voter server-317853c2-4e22-8096-56af-6e5b6339a79f 317853c2-4e22-8096-56af-6e5b6339a79f}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2025-06-26T05:13:09.354Z [INFO]  server-51fcdc09-c66b-6312-09df-794bbb495d3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-51fcdc09-c66b-6312-09df-794bbb495d3b Address:51fcdc09-c66b-6312-09df-794bbb495d3b} {Suffrage:Voter ID:server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3 Address:8722c6c2-0e5f-a963-dacc-40a2a5cec4e3} {Suffrage:Voter ID:server-1f6a2d25-248d-e0e7-5633-3871c0144bed Address:1f6a2d25-248d-e0e7-5633-3871c0144bed}]"
2025-06-26T05:13:09.354Z [INFO]  server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-51fcdc09-c66b-6312-09df-794bbb495d3b Address:51fcdc09-c66b-6312-09df-794bbb495d3b} {Suffrage:Voter ID:server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3 Address:8722c6c2-0e5f-a963-dacc-40a2a5cec4e3} {Suffrage:Voter ID:server-1f6a2d25-248d-e0e7-5633-3871c0144bed Address:1f6a2d25-248d-e0e7-5633-3871c0144bed}]"
2025-06-26T05:13:09.354Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-51fcdc09-c66b-6312-09df-794bbb495d3b Address:51fcdc09-c66b-6312-09df-794bbb495d3b} {Suffrage:Voter ID:server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3 Address:8722c6c2-0e5f-a963-dacc-40a2a5cec4e3} {Suffrage:Voter ID:server-1f6a2d25-248d-e0e7-5633-3871c0144bed Address:1f6a2d25-248d-e0e7-5633-3871c0144bed}]"
2025-06-26T05:13:09.354Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:09.354Z [INFO]  server-51fcdc09-c66b-6312-09df-794bbb495d3b: entering follower state: follower="Node at 51fcdc09-c66b-6312-09df-794bbb495d3b [Follower]" leader-address= leader-id=
2025-06-26T05:13:09.354Z [INFO]  server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3: entering follower state: follower="Node at 8722c6c2-0e5f-a963-dacc-40a2a5cec4e3 [Follower]" leader-address= leader-id=
2025-06-26T05:13:09.354Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: entering follower state: follower="Node at 1f6a2d25-248d-e0e7-5633-3871c0144bed [Follower]" leader-address= leader-id=
2025-06-26T05:13:09.917Z [WARN]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:09.917Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: entering candidate state: node="Node at 1f6a2d25-248d-e0e7-5633-3871c0144bed [Candidate]" term=2
2025-06-26T05:13:09.917Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: election won: term=2 tally=2
2025-06-26T05:13:09.917Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: entering leader state: leader="Node at 1f6a2d25-248d-e0e7-5633-3871c0144bed [Leader]"
2025-06-26T05:13:09.917Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: added peer, starting replication: peer=server-51fcdc09-c66b-6312-09df-794bbb495d3b
2025-06-26T05:13:09.917Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: added peer, starting replication: peer=server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3
2025-06-26T05:13:09.917Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: pipelining replication: peer="{Voter server-51fcdc09-c66b-6312-09df-794bbb495d3b 51fcdc09-c66b-6312-09df-794bbb495d3b}"
2025-06-26T05:13:09.917Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: pipelining replication: peer="{Voter server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3 8722c6c2-0e5f-a963-dacc-40a2a5cec4e3}"
2025-06-26T05:13:10.925Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:09.917899243 +0000 UTC m=+18.067009521 (1 nodes), 563.50708ms from start of poll, 563.6932ms from cluster start. Timeout at 2025-06-26 05:13:10.925679203 +0000 UTC m=+19.074789521, 1.00778s after stability
2025-06-26T05:13:10.926Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: starting snapshot up to: index=12
2025-06-26T05:13:10.926Z [INFO]  creating new snapshot: path=/tmp/raft2975392574/snapshots/2-12-1750914790926.tmp
2025-06-26T05:13:10.926Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: snapshot complete up to: index=12
2025-06-26T05:13:10.926Z [INFO]  creating new snapshot: path=/tmp/raft2975392574/snapshots/2-23-1750914790926.tmp
2025-06-26T05:13:10.927Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: copied to local snapshot: bytes=71
2025-06-26T05:13:10.927Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: snapshot restore progress: id=2-23-1750914790926 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:10.927Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: removing all old logs from log store
2025-06-26T05:13:10.927Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: compacting logs: from=1 to=22
2025-06-26T05:13:10.927Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: restored user snapshot: index=23
2025-06-26T05:13:10.927Z [ERROR] server-1f6a2d25-248d-e0e7-5633-3871c0144bed: failed to get log: index=22 error="log not found"
2025-06-26T05:13:10.927Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: aborting pipeline replication: peer="{Voter server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3 8722c6c2-0e5f-a963-dacc-40a2a5cec4e3}"
2025-06-26T05:13:10.927Z [ERROR] server-1f6a2d25-248d-e0e7-5633-3871c0144bed: failed to get log: index=22 error="log not found"
2025-06-26T05:13:10.927Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: aborting pipeline replication: peer="{Voter server-51fcdc09-c66b-6312-09df-794bbb495d3b 51fcdc09-c66b-6312-09df-794bbb495d3b}"
2025-06-26T05:13:10.933Z [ERROR] server-1f6a2d25-248d-e0e7-5633-3871c0144bed: failed to get log: index=22 error="log not found"
2025-06-26T05:13:10.933Z [ERROR] server-1f6a2d25-248d-e0e7-5633-3871c0144bed: failed to get log: index=22 error="log not found"
2025-06-26T05:13:10.933Z [INFO]  creating new snapshot: path=/tmp/raft494891736/snapshots/2-23-1750914790933.tmp
2025-06-26T05:13:10.933Z [INFO]  server-51fcdc09-c66b-6312-09df-794bbb495d3b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:10.933Z [INFO]  creating new snapshot: path=/tmp/raft1338829109/snapshots/2-23-1750914790933.tmp
2025-06-26T05:13:10.933Z [INFO]  server-51fcdc09-c66b-6312-09df-794bbb495d3b: copied to local snapshot: bytes=71
2025-06-26T05:13:10.933Z [INFO]  server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:10.934Z [INFO]  server-51fcdc09-c66b-6312-09df-794bbb495d3b: snapshot restore progress: id=2-23-1750914790933 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:10.934Z [INFO]  server-51fcdc09-c66b-6312-09df-794bbb495d3b: removing all old logs from log store
2025-06-26T05:13:10.934Z [INFO]  server-51fcdc09-c66b-6312-09df-794bbb495d3b: compacting logs: from=1 to=22
2025-06-26T05:13:10.934Z [INFO]  server-51fcdc09-c66b-6312-09df-794bbb495d3b: Installed remote snapshot
2025-06-26T05:13:10.934Z [INFO]  server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3: copied to local snapshot: bytes=71
2025-06-26T05:13:10.934Z [WARN]  server-51fcdc09-c66b-6312-09df-794bbb495d3b: skipping application of old log: index=22
2025-06-26T05:13:10.934Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: pipelining replication: peer="{Voter server-51fcdc09-c66b-6312-09df-794bbb495d3b 51fcdc09-c66b-6312-09df-794bbb495d3b}"
2025-06-26T05:13:10.934Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:10.934Z [INFO]  server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3: snapshot restore progress: id=2-23-1750914790933 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:10.934Z [INFO]  server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3: removing all old logs from log store
2025-06-26T05:13:10.934Z [INFO]  server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3: compacting logs: from=1 to=22
2025-06-26T05:13:10.934Z [INFO]  server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3: Installed remote snapshot
2025-06-26T05:13:10.934Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: pipelining replication: peer="{Voter server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3 8722c6c2-0e5f-a963-dacc-40a2a5cec4e3}"
2025-06-26T05:13:11.941Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:10.934180563 +0000 UTC m=+19.083290841 (2 nodes), 480ns from start of poll, 1.57997452s from cluster start. Timeout at 2025-06-26 05:13:11.941606643 +0000 UTC m=+20.090716921, 1.00742608s after stability
2025-06-26T05:13:11.957Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: aborting pipeline replication: peer="{Voter server-8722c6c2-0e5f-a963-dacc-40a2a5cec4e3 8722c6c2-0e5f-a963-dacc-40a2a5cec4e3}"
2025-06-26T05:13:11.957Z [INFO]  server-1f6a2d25-248d-e0e7-5633-3871c0144bed: aborting pipeline replication: peer="{Voter server-51fcdc09-c66b-6312-09df-794bbb495d3b 51fcdc09-c66b-6312-09df-794bbb495d3b}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2025-06-26T05:13:11.958Z [INFO]  server-229ea9e7-5393-2627-4199-860635ee8272: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-229ea9e7-5393-2627-4199-860635ee8272 Address:229ea9e7-5393-2627-4199-860635ee8272} {Suffrage:Voter ID:server-659368b4-a099-6384-e7ef-adec2b57988d Address:659368b4-a099-6384-e7ef-adec2b57988d} {Suffrage:Voter ID:server-0afe31d4-ad79-6732-477e-31856b3fc450 Address:0afe31d4-ad79-6732-477e-31856b3fc450}]"
2025-06-26T05:13:11.958Z [INFO]  server-659368b4-a099-6384-e7ef-adec2b57988d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-229ea9e7-5393-2627-4199-860635ee8272 Address:229ea9e7-5393-2627-4199-860635ee8272} {Suffrage:Voter ID:server-659368b4-a099-6384-e7ef-adec2b57988d Address:659368b4-a099-6384-e7ef-adec2b57988d} {Suffrage:Voter ID:server-0afe31d4-ad79-6732-477e-31856b3fc450 Address:0afe31d4-ad79-6732-477e-31856b3fc450}]"
2025-06-26T05:13:11.958Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-229ea9e7-5393-2627-4199-860635ee8272 Address:229ea9e7-5393-2627-4199-860635ee8272} {Suffrage:Voter ID:server-659368b4-a099-6384-e7ef-adec2b57988d Address:659368b4-a099-6384-e7ef-adec2b57988d} {Suffrage:Voter ID:server-0afe31d4-ad79-6732-477e-31856b3fc450 Address:0afe31d4-ad79-6732-477e-31856b3fc450}]"
2025-06-26T05:13:11.958Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:11.958Z [INFO]  server-229ea9e7-5393-2627-4199-860635ee8272: entering follower state: follower="Node at 229ea9e7-5393-2627-4199-860635ee8272 [Follower]" leader-address= leader-id=
2025-06-26T05:13:11.958Z [INFO]  server-659368b4-a099-6384-e7ef-adec2b57988d: entering follower state: follower="Node at 659368b4-a099-6384-e7ef-adec2b57988d [Follower]" leader-address= leader-id=
2025-06-26T05:13:11.958Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: entering follower state: follower="Node at 0afe31d4-ad79-6732-477e-31856b3fc450 [Follower]" leader-address= leader-id=
2025-06-26T05:13:12.505Z [WARN]  server-0afe31d4-ad79-6732-477e-31856b3fc450: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:12.505Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: entering candidate state: node="Node at 0afe31d4-ad79-6732-477e-31856b3fc450 [Candidate]" term=2
2025-06-26T05:13:12.505Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: election won: term=2 tally=2
2025-06-26T05:13:12.505Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: entering leader state: leader="Node at 0afe31d4-ad79-6732-477e-31856b3fc450 [Leader]"
2025-06-26T05:13:12.505Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: added peer, starting replication: peer=server-229ea9e7-5393-2627-4199-860635ee8272
2025-06-26T05:13:12.505Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: added peer, starting replication: peer=server-659368b4-a099-6384-e7ef-adec2b57988d
2025-06-26T05:13:12.505Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: pipelining replication: peer="{Voter server-659368b4-a099-6384-e7ef-adec2b57988d 659368b4-a099-6384-e7ef-adec2b57988d}"
2025-06-26T05:13:12.505Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: pipelining replication: peer="{Voter server-229ea9e7-5393-2627-4199-860635ee8272 229ea9e7-5393-2627-4199-860635ee8272}"
2025-06-26T05:13:13.513Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:12.505976843 +0000 UTC m=+20.655087121 (1 nodes), 547.59604ms from start of poll, 547.78736ms from cluster start. Timeout at 2025-06-26 05:13:13.513713763 +0000 UTC m=+21.662824041, 1.00773692s after stability
2025-06-26T05:13:13.514Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: starting snapshot up to: index=12
2025-06-26T05:13:13.514Z [INFO]  creating new snapshot: path=/tmp/raft3146705865/snapshots/2-12-1750914793514.tmp
2025-06-26T05:13:13.514Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: snapshot complete up to: index=12
2025-06-26T05:13:13.515Z [INFO]  server-3e480903-c8d6-f75b-4558-be84c4a3e9e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e480903-c8d6-f75b-4558-be84c4a3e9e5 Address:3e480903-c8d6-f75b-4558-be84c4a3e9e5} {Suffrage:Voter ID:server-7572a1aa-1bf6-637b-adcf-163a03444e23 Address:7572a1aa-1bf6-637b-adcf-163a03444e23} {Suffrage:Voter ID:server-37193216-eb8c-a2a4-f32d-e72b6296f1f9 Address:37193216-eb8c-a2a4-f32d-e72b6296f1f9}]"
2025-06-26T05:13:13.515Z [INFO]  server-7572a1aa-1bf6-637b-adcf-163a03444e23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e480903-c8d6-f75b-4558-be84c4a3e9e5 Address:3e480903-c8d6-f75b-4558-be84c4a3e9e5} {Suffrage:Voter ID:server-7572a1aa-1bf6-637b-adcf-163a03444e23 Address:7572a1aa-1bf6-637b-adcf-163a03444e23} {Suffrage:Voter ID:server-37193216-eb8c-a2a4-f32d-e72b6296f1f9 Address:37193216-eb8c-a2a4-f32d-e72b6296f1f9}]"
2025-06-26T05:13:13.515Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e480903-c8d6-f75b-4558-be84c4a3e9e5 Address:3e480903-c8d6-f75b-4558-be84c4a3e9e5} {Suffrage:Voter ID:server-7572a1aa-1bf6-637b-adcf-163a03444e23 Address:7572a1aa-1bf6-637b-adcf-163a03444e23} {Suffrage:Voter ID:server-37193216-eb8c-a2a4-f32d-e72b6296f1f9 Address:37193216-eb8c-a2a4-f32d-e72b6296f1f9}]"
2025-06-26T05:13:13.515Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:13.515Z [INFO]  server-3e480903-c8d6-f75b-4558-be84c4a3e9e5: entering follower state: follower="Node at 3e480903-c8d6-f75b-4558-be84c4a3e9e5 [Follower]" leader-address= leader-id=
2025-06-26T05:13:13.515Z [INFO]  server-7572a1aa-1bf6-637b-adcf-163a03444e23: entering follower state: follower="Node at 7572a1aa-1bf6-637b-adcf-163a03444e23 [Follower]" leader-address= leader-id=
2025-06-26T05:13:13.515Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: entering follower state: follower="Node at 37193216-eb8c-a2a4-f32d-e72b6296f1f9 [Follower]" leader-address= leader-id=
2025-06-26T05:13:14.269Z [WARN]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:14.269Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: entering candidate state: node="Node at 37193216-eb8c-a2a4-f32d-e72b6296f1f9 [Candidate]" term=2
2025-06-26T05:13:14.269Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: election won: term=2 tally=2
2025-06-26T05:13:14.269Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: entering leader state: leader="Node at 37193216-eb8c-a2a4-f32d-e72b6296f1f9 [Leader]"
2025-06-26T05:13:14.269Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: added peer, starting replication: peer=server-3e480903-c8d6-f75b-4558-be84c4a3e9e5
2025-06-26T05:13:14.269Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: added peer, starting replication: peer=server-7572a1aa-1bf6-637b-adcf-163a03444e23
2025-06-26T05:13:14.269Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: pipelining replication: peer="{Voter server-7572a1aa-1bf6-637b-adcf-163a03444e23 7572a1aa-1bf6-637b-adcf-163a03444e23}"
2025-06-26T05:13:14.270Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: pipelining replication: peer="{Voter server-3e480903-c8d6-f75b-4558-be84c4a3e9e5 3e480903-c8d6-f75b-4558-be84c4a3e9e5}"
2025-06-26T05:13:15.277Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:14.270059483 +0000 UTC m=+22.419169721 (1 nodes), 754.85468ms from start of poll, 755.06704ms from cluster start. Timeout at 2025-06-26 05:13:15.277719523 +0000 UTC m=+23.426829801, 1.00766008s after stability
2025-06-26T05:13:15.277Z [INFO]  creating new snapshot: path=/tmp/raft3767735162/snapshots/2-13-1750914795277.tmp
2025-06-26T05:13:15.278Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: copied to local snapshot: bytes=71
2025-06-26T05:13:15.278Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: snapshot restore progress: id=2-13-1750914795277 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:15.278Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: restored user snapshot: index=13
2025-06-26T05:13:15.278Z [ERROR] server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: failed to get log: index=3 error="log not found"
2025-06-26T05:13:15.278Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: aborting pipeline replication: peer="{Voter server-7572a1aa-1bf6-637b-adcf-163a03444e23 7572a1aa-1bf6-637b-adcf-163a03444e23}"
2025-06-26T05:13:15.278Z [ERROR] server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: failed to get log: index=3 error="log not found"
2025-06-26T05:13:15.278Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: aborting pipeline replication: peer="{Voter server-3e480903-c8d6-f75b-4558-be84c4a3e9e5 3e480903-c8d6-f75b-4558-be84c4a3e9e5}"
2025-06-26T05:13:15.285Z [ERROR] server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: failed to get log: index=3 error="log not found"
2025-06-26T05:13:15.285Z [ERROR] server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: failed to get log: index=3 error="log not found"
2025-06-26T05:13:15.285Z [INFO]  creating new snapshot: path=/tmp/raft3248116529/snapshots/2-13-1750914795285.tmp
2025-06-26T05:13:15.285Z [INFO]  creating new snapshot: path=/tmp/raft1990871549/snapshots/2-13-1750914795285.tmp
2025-06-26T05:13:15.286Z [INFO]  server-7572a1aa-1bf6-637b-adcf-163a03444e23: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:15.286Z [INFO]  server-3e480903-c8d6-f75b-4558-be84c4a3e9e5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:15.286Z [INFO]  server-7572a1aa-1bf6-637b-adcf-163a03444e23: copied to local snapshot: bytes=71
2025-06-26T05:13:15.286Z [INFO]  server-3e480903-c8d6-f75b-4558-be84c4a3e9e5: copied to local snapshot: bytes=71
2025-06-26T05:13:15.286Z [INFO]  server-7572a1aa-1bf6-637b-adcf-163a03444e23: snapshot restore progress: id=2-13-1750914795285 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:15.286Z [INFO]  server-7572a1aa-1bf6-637b-adcf-163a03444e23: Installed remote snapshot
2025-06-26T05:13:15.286Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: pipelining replication: peer="{Voter server-7572a1aa-1bf6-637b-adcf-163a03444e23 7572a1aa-1bf6-637b-adcf-163a03444e23}"
2025-06-26T05:13:15.286Z [INFO]  server-3e480903-c8d6-f75b-4558-be84c4a3e9e5: snapshot restore progress: id=2-13-1750914795285 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:15.286Z [INFO]  server-3e480903-c8d6-f75b-4558-be84c4a3e9e5: Installed remote snapshot
2025-06-26T05:13:15.286Z [INFO]  server-37193216-eb8c-a2a4-f32d-e72b6296f1f9: pipelining replication: peer="{Voter server-3e480903-c8d6-f75b-4558-be84c4a3e9e5 3e480903-c8d6-f75b-4558-be84c4a3e9e5}"
2025-06-26T05:13:15.286Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:16.293Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:15.286494123 +0000 UTC m=+23.435604361 (2 nodes), 360ns from start of poll, 1.77150168s from cluster start. Timeout at 2025-06-26 05:13:16.293800603 +0000 UTC m=+24.442910881, 1.00730652s after stability
2025-06-26T05:13:16.309Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: aborting pipeline replication: peer="{Voter server-659368b4-a099-6384-e7ef-adec2b57988d 659368b4-a099-6384-e7ef-adec2b57988d}"
2025-06-26T05:13:16.309Z [INFO]  server-0afe31d4-ad79-6732-477e-31856b3fc450: aborting pipeline replication: peer="{Voter server-229ea9e7-5393-2627-4199-860635ee8272 229ea9e7-5393-2627-4199-860635ee8272}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2025-06-26T05:13:16.310Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1740f29-696f-503e-6352-4ee67cf217bc Address:a1740f29-696f-503e-6352-4ee67cf217bc} {Suffrage:Voter ID:server-c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10 Address:c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10} {Suffrage:Voter ID:server-cc018aa9-e44a-ad03-5ff2-196fb658e1ea Address:cc018aa9-e44a-ad03-5ff2-196fb658e1ea}]"
2025-06-26T05:13:16.310Z [INFO]  server-c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1740f29-696f-503e-6352-4ee67cf217bc Address:a1740f29-696f-503e-6352-4ee67cf217bc} {Suffrage:Voter ID:server-c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10 Address:c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10} {Suffrage:Voter ID:server-cc018aa9-e44a-ad03-5ff2-196fb658e1ea Address:cc018aa9-e44a-ad03-5ff2-196fb658e1ea}]"
2025-06-26T05:13:16.310Z [INFO]  server-cc018aa9-e44a-ad03-5ff2-196fb658e1ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1740f29-696f-503e-6352-4ee67cf217bc Address:a1740f29-696f-503e-6352-4ee67cf217bc} {Suffrage:Voter ID:server-c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10 Address:c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10} {Suffrage:Voter ID:server-cc018aa9-e44a-ad03-5ff2-196fb658e1ea Address:cc018aa9-e44a-ad03-5ff2-196fb658e1ea}]"
2025-06-26T05:13:16.310Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:16.310Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: entering follower state: follower="Node at a1740f29-696f-503e-6352-4ee67cf217bc [Follower]" leader-address= leader-id=
2025-06-26T05:13:16.310Z [INFO]  server-c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10: entering follower state: follower="Node at c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10 [Follower]" leader-address= leader-id=
2025-06-26T05:13:16.310Z [INFO]  server-cc018aa9-e44a-ad03-5ff2-196fb658e1ea: entering follower state: follower="Node at cc018aa9-e44a-ad03-5ff2-196fb658e1ea [Follower]" leader-address= leader-id=
2025-06-26T05:13:16.997Z [WARN]  server-a1740f29-696f-503e-6352-4ee67cf217bc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:16.997Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: entering candidate state: node="Node at a1740f29-696f-503e-6352-4ee67cf217bc [Candidate]" term=2
2025-06-26T05:13:16.997Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: election won: term=2 tally=2
2025-06-26T05:13:16.997Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: entering leader state: leader="Node at a1740f29-696f-503e-6352-4ee67cf217bc [Leader]"
2025-06-26T05:13:16.997Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: added peer, starting replication: peer=server-c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10
2025-06-26T05:13:16.997Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: added peer, starting replication: peer=server-cc018aa9-e44a-ad03-5ff2-196fb658e1ea
2025-06-26T05:13:16.997Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: pipelining replication: peer="{Voter server-cc018aa9-e44a-ad03-5ff2-196fb658e1ea cc018aa9-e44a-ad03-5ff2-196fb658e1ea}"
2025-06-26T05:13:16.997Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: pipelining replication: peer="{Voter server-c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10 c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10}"
2025-06-26T05:13:18.005Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:16.998022803 +0000 UTC m=+25.147133081 (1 nodes), 687.72192ms from start of poll, 687.91004ms from cluster start. Timeout at 2025-06-26 05:13:18.005604283 +0000 UTC m=+26.154714561, 1.00758148s after stability
2025-06-26T05:13:18.006Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: starting snapshot up to: index=12
2025-06-26T05:13:18.006Z [INFO]  creating new snapshot: path=/tmp/raft364367842/snapshots/2-12-1750914798006.tmp
2025-06-26T05:13:18.006Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: snapshot complete up to: index=12
2025-06-26T05:13:18.007Z [INFO]  server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1 Address:fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1} {Suffrage:Voter ID:server-facf1708-279c-4d5d-fc5e-a373be6c836e Address:facf1708-279c-4d5d-fc5e-a373be6c836e} {Suffrage:Voter ID:server-e5c49da7-17d3-8771-ca67-4b4d70fe3331 Address:e5c49da7-17d3-8771-ca67-4b4d70fe3331}]"
2025-06-26T05:13:18.007Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1 Address:fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1} {Suffrage:Voter ID:server-facf1708-279c-4d5d-fc5e-a373be6c836e Address:facf1708-279c-4d5d-fc5e-a373be6c836e} {Suffrage:Voter ID:server-e5c49da7-17d3-8771-ca67-4b4d70fe3331 Address:e5c49da7-17d3-8771-ca67-4b4d70fe3331}]"
2025-06-26T05:13:18.007Z [INFO]  server-e5c49da7-17d3-8771-ca67-4b4d70fe3331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1 Address:fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1} {Suffrage:Voter ID:server-facf1708-279c-4d5d-fc5e-a373be6c836e Address:facf1708-279c-4d5d-fc5e-a373be6c836e} {Suffrage:Voter ID:server-e5c49da7-17d3-8771-ca67-4b4d70fe3331 Address:e5c49da7-17d3-8771-ca67-4b4d70fe3331}]"
2025-06-26T05:13:18.007Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:18.007Z [INFO]  server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1: entering follower state: follower="Node at fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1 [Follower]" leader-address= leader-id=
2025-06-26T05:13:18.007Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: entering follower state: follower="Node at facf1708-279c-4d5d-fc5e-a373be6c836e [Follower]" leader-address= leader-id=
2025-06-26T05:13:18.007Z [INFO]  server-e5c49da7-17d3-8771-ca67-4b4d70fe3331: entering follower state: follower="Node at e5c49da7-17d3-8771-ca67-4b4d70fe3331 [Follower]" leader-address= leader-id=
2025-06-26T05:13:18.561Z [WARN]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:18.561Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: entering candidate state: node="Node at facf1708-279c-4d5d-fc5e-a373be6c836e [Candidate]" term=2
2025-06-26T05:13:18.561Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: election won: term=2 tally=2
2025-06-26T05:13:18.561Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: entering leader state: leader="Node at facf1708-279c-4d5d-fc5e-a373be6c836e [Leader]"
2025-06-26T05:13:18.561Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: added peer, starting replication: peer=server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1
2025-06-26T05:13:18.561Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: added peer, starting replication: peer=server-e5c49da7-17d3-8771-ca67-4b4d70fe3331
2025-06-26T05:13:18.561Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: pipelining replication: peer="{Voter server-e5c49da7-17d3-8771-ca67-4b4d70fe3331 e5c49da7-17d3-8771-ca67-4b4d70fe3331}"
2025-06-26T05:13:18.561Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: pipelining replication: peer="{Voter server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1 fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1}"
2025-06-26T05:13:19.569Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:18.562014603 +0000 UTC m=+26.711124881 (1 nodes), 554.81584ms from start of poll, 555.02412ms from cluster start. Timeout at 2025-06-26 05:13:19.569604523 +0000 UTC m=+27.718714801, 1.00758992s after stability
2025-06-26T05:13:19.569Z [INFO]  creating new snapshot: path=/tmp/raft1291313828/snapshots/2-13-1750914799569.tmp
2025-06-26T05:13:19.570Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: copied to local snapshot: bytes=71
2025-06-26T05:13:19.570Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: snapshot restore progress: id=2-13-1750914799569 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:19.570Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: removing all old logs from log store
2025-06-26T05:13:19.570Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: compacting logs: from=1 to=2
2025-06-26T05:13:19.570Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: restored user snapshot: index=13
2025-06-26T05:13:19.570Z [ERROR] server-facf1708-279c-4d5d-fc5e-a373be6c836e: failed to get log: index=2 error="log not found"
2025-06-26T05:13:19.570Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: aborting pipeline replication: peer="{Voter server-e5c49da7-17d3-8771-ca67-4b4d70fe3331 e5c49da7-17d3-8771-ca67-4b4d70fe3331}"
2025-06-26T05:13:19.570Z [ERROR] server-facf1708-279c-4d5d-fc5e-a373be6c836e: failed to get log: index=2 error="log not found"
2025-06-26T05:13:19.570Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: aborting pipeline replication: peer="{Voter server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1 fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1}"
2025-06-26T05:13:19.577Z [ERROR] server-facf1708-279c-4d5d-fc5e-a373be6c836e: failed to get log: index=2 error="log not found"
2025-06-26T05:13:19.577Z [INFO]  creating new snapshot: path=/tmp/raft2835826971/snapshots/2-13-1750914799577.tmp
2025-06-26T05:13:19.578Z [INFO]  server-e5c49da7-17d3-8771-ca67-4b4d70fe3331: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:19.578Z [INFO]  server-e5c49da7-17d3-8771-ca67-4b4d70fe3331: copied to local snapshot: bytes=71
2025-06-26T05:13:19.578Z [INFO]  server-e5c49da7-17d3-8771-ca67-4b4d70fe3331: snapshot restore progress: id=2-13-1750914799577 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:19.578Z [INFO]  server-e5c49da7-17d3-8771-ca67-4b4d70fe3331: removing all old logs from log store
2025-06-26T05:13:19.578Z [INFO]  server-e5c49da7-17d3-8771-ca67-4b4d70fe3331: compacting logs: from=1 to=2
2025-06-26T05:13:19.578Z [INFO]  server-e5c49da7-17d3-8771-ca67-4b4d70fe3331: Installed remote snapshot
2025-06-26T05:13:19.578Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: pipelining replication: peer="{Voter server-e5c49da7-17d3-8771-ca67-4b4d70fe3331 e5c49da7-17d3-8771-ca67-4b4d70fe3331}"
2025-06-26T05:13:19.578Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:19.581Z [ERROR] server-facf1708-279c-4d5d-fc5e-a373be6c836e: failed to get log: index=2 error="log not found"
2025-06-26T05:13:19.581Z [INFO]  creating new snapshot: path=/tmp/raft2112306006/snapshots/2-13-1750914799581.tmp
2025-06-26T05:13:19.581Z [INFO]  server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:19.581Z [INFO]  server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1: copied to local snapshot: bytes=71
2025-06-26T05:13:19.582Z [INFO]  server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1: snapshot restore progress: id=2-13-1750914799581 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:19.582Z [INFO]  server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1: removing all old logs from log store
2025-06-26T05:13:19.582Z [INFO]  server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1: compacting logs: from=1 to=2
2025-06-26T05:13:19.582Z [INFO]  server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1: Installed remote snapshot
2025-06-26T05:13:19.582Z [INFO]  server-facf1708-279c-4d5d-fc5e-a373be6c836e: pipelining replication: peer="{Voter server-fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1 fda8fc3a-49b3-908e-b45c-2c4cb74eb5b1}"
2025-06-26T05:13:20.585Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:19.578453523 +0000 UTC m=+27.727563801 (2 nodes), 880ns from start of poll, 1.57146304s from cluster start. Timeout at 2025-06-26 05:13:20.585589923 +0000 UTC m=+28.734700201, 1.0071364s after stability
2025-06-26T05:13:20.609Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: aborting pipeline replication: peer="{Voter server-cc018aa9-e44a-ad03-5ff2-196fb658e1ea cc018aa9-e44a-ad03-5ff2-196fb658e1ea}"
2025-06-26T05:13:20.610Z [INFO]  server-a1740f29-696f-503e-6352-4ee67cf217bc: aborting pipeline replication: peer="{Voter server-c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10 c8b3c3f7-6fd3-a2cb-7c57-4838b1bfef10}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2025-06-26T05:13:20.610Z [INFO]  server-3f28d93b-e56f-7553-8058-8b6906fa1a97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f28d93b-e56f-7553-8058-8b6906fa1a97 Address:3f28d93b-e56f-7553-8058-8b6906fa1a97} {Suffrage:Voter ID:server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf Address:e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf} {Suffrage:Voter ID:server-76935ea7-7197-77ad-55fc-70e64ce4e310 Address:76935ea7-7197-77ad-55fc-70e64ce4e310}]"
2025-06-26T05:13:20.610Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f28d93b-e56f-7553-8058-8b6906fa1a97 Address:3f28d93b-e56f-7553-8058-8b6906fa1a97} {Suffrage:Voter ID:server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf Address:e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf} {Suffrage:Voter ID:server-76935ea7-7197-77ad-55fc-70e64ce4e310 Address:76935ea7-7197-77ad-55fc-70e64ce4e310}]"
2025-06-26T05:13:20.610Z [INFO]  server-76935ea7-7197-77ad-55fc-70e64ce4e310: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f28d93b-e56f-7553-8058-8b6906fa1a97 Address:3f28d93b-e56f-7553-8058-8b6906fa1a97} {Suffrage:Voter ID:server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf Address:e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf} {Suffrage:Voter ID:server-76935ea7-7197-77ad-55fc-70e64ce4e310 Address:76935ea7-7197-77ad-55fc-70e64ce4e310}]"
2025-06-26T05:13:20.610Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:20.629Z [INFO]  server-3f28d93b-e56f-7553-8058-8b6906fa1a97: entering follower state: follower="Node at 3f28d93b-e56f-7553-8058-8b6906fa1a97 [Follower]" leader-address= leader-id=
2025-06-26T05:13:20.629Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: entering follower state: follower="Node at e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf [Follower]" leader-address= leader-id=
2025-06-26T05:13:20.629Z [INFO]  server-76935ea7-7197-77ad-55fc-70e64ce4e310: entering follower state: follower="Node at 76935ea7-7197-77ad-55fc-70e64ce4e310 [Follower]" leader-address= leader-id=
2025-06-26T05:13:21.145Z [WARN]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:21.145Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: entering candidate state: node="Node at e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf [Candidate]" term=2
2025-06-26T05:13:21.145Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: election won: term=2 tally=2
2025-06-26T05:13:21.145Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: entering leader state: leader="Node at e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf [Leader]"
2025-06-26T05:13:21.145Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: added peer, starting replication: peer=server-3f28d93b-e56f-7553-8058-8b6906fa1a97
2025-06-26T05:13:21.145Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: added peer, starting replication: peer=server-76935ea7-7197-77ad-55fc-70e64ce4e310
2025-06-26T05:13:21.145Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: pipelining replication: peer="{Voter server-76935ea7-7197-77ad-55fc-70e64ce4e310 76935ea7-7197-77ad-55fc-70e64ce4e310}"
2025-06-26T05:13:21.145Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: pipelining replication: peer="{Voter server-3f28d93b-e56f-7553-8058-8b6906fa1a97 3f28d93b-e56f-7553-8058-8b6906fa1a97}"
2025-06-26T05:13:22.153Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:21.146031803 +0000 UTC m=+29.295142081 (1 nodes), 535.35792ms from start of poll, 535.5864ms from cluster start. Timeout at 2025-06-26 05:13:22.153605963 +0000 UTC m=+30.302716641, 1.00757456s after stability
2025-06-26T05:13:22.154Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: starting snapshot up to: index=12
2025-06-26T05:13:22.154Z [INFO]  creating new snapshot: path=/tmp/raft2380110253/snapshots/2-12-1750914802154.tmp
2025-06-26T05:13:22.154Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: snapshot complete up to: index=12
2025-06-26T05:13:22.154Z [INFO]  creating new snapshot: path=/tmp/raft2380110253/snapshots/2-23-1750914802154.tmp
2025-06-26T05:13:22.155Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: copied to local snapshot: bytes=71
2025-06-26T05:13:22.155Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: snapshot restore progress: id=2-23-1750914802154 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:22.155Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: restored user snapshot: index=23
2025-06-26T05:13:22.155Z [ERROR] server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: failed to get log: index=23 error="log not found"
2025-06-26T05:13:22.155Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: aborting pipeline replication: peer="{Voter server-76935ea7-7197-77ad-55fc-70e64ce4e310 76935ea7-7197-77ad-55fc-70e64ce4e310}"
2025-06-26T05:13:22.155Z [ERROR] server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: failed to get log: index=23 error="log not found"
2025-06-26T05:13:22.155Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: aborting pipeline replication: peer="{Voter server-3f28d93b-e56f-7553-8058-8b6906fa1a97 3f28d93b-e56f-7553-8058-8b6906fa1a97}"
2025-06-26T05:13:22.165Z [ERROR] server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: failed to get log: index=23 error="log not found"
2025-06-26T05:13:22.165Z [ERROR] server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: failed to get log: index=23 error="log not found"
2025-06-26T05:13:22.165Z [INFO]  creating new snapshot: path=/tmp/raft445110793/snapshots/2-23-1750914802165.tmp
2025-06-26T05:13:22.166Z [INFO]  creating new snapshot: path=/tmp/raft3142890885/snapshots/2-23-1750914802166.tmp
2025-06-26T05:13:22.166Z [INFO]  server-76935ea7-7197-77ad-55fc-70e64ce4e310: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:22.166Z [INFO]  server-3f28d93b-e56f-7553-8058-8b6906fa1a97: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:22.166Z [INFO]  server-76935ea7-7197-77ad-55fc-70e64ce4e310: copied to local snapshot: bytes=71
2025-06-26T05:13:22.166Z [INFO]  server-76935ea7-7197-77ad-55fc-70e64ce4e310: snapshot restore progress: id=2-23-1750914802165 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:22.166Z [INFO]  server-76935ea7-7197-77ad-55fc-70e64ce4e310: Installed remote snapshot
2025-06-26T05:13:22.166Z [WARN]  server-76935ea7-7197-77ad-55fc-70e64ce4e310: skipping application of old log: index=22
2025-06-26T05:13:22.166Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: pipelining replication: peer="{Voter server-76935ea7-7197-77ad-55fc-70e64ce4e310 76935ea7-7197-77ad-55fc-70e64ce4e310}"
2025-06-26T05:13:22.166Z [INFO]  server-3f28d93b-e56f-7553-8058-8b6906fa1a97: copied to local snapshot: bytes=71
2025-06-26T05:13:22.166Z [INFO]  server-3f28d93b-e56f-7553-8058-8b6906fa1a97: snapshot restore progress: id=2-23-1750914802166 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:22.166Z [INFO]  server-3f28d93b-e56f-7553-8058-8b6906fa1a97: Installed remote snapshot
2025-06-26T05:13:22.166Z [WARN]  server-3f28d93b-e56f-7553-8058-8b6906fa1a97: skipping application of old log: index=22
2025-06-26T05:13:22.166Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: pipelining replication: peer="{Voter server-3f28d93b-e56f-7553-8058-8b6906fa1a97 3f28d93b-e56f-7553-8058-8b6906fa1a97}"
2025-06-26T05:13:22.166Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:23.173Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:22.166699603 +0000 UTC m=+30.315809881 (2 nodes), 680ns from start of poll, 1.5562542s from cluster start. Timeout at 2025-06-26 05:13:23.173718083 +0000 UTC m=+31.322828361, 1.00701848s after stability
2025-06-26T05:13:23.189Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: aborting pipeline replication: peer="{Voter server-76935ea7-7197-77ad-55fc-70e64ce4e310 76935ea7-7197-77ad-55fc-70e64ce4e310}"
2025-06-26T05:13:23.189Z [INFO]  server-e19d90a3-b681-8a22-4c9d-3be0b0d7a7bf: aborting pipeline replication: peer="{Voter server-3f28d93b-e56f-7553-8058-8b6906fa1a97 3f28d93b-e56f-7553-8058-8b6906fa1a97}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2025-06-26T05:13:23.190Z [INFO]  server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2 Address:ca34c9e9-77b8-12d7-8c40-3462bfddffe2} {Suffrage:Voter ID:server-aceceea1-a510-652f-7d54-fc9bd7dd54db Address:aceceea1-a510-652f-7d54-fc9bd7dd54db} {Suffrage:Voter ID:server-16088dbc-1f08-4746-53fb-f9bc85878227 Address:16088dbc-1f08-4746-53fb-f9bc85878227}]"
2025-06-26T05:13:23.190Z [INFO]  server-aceceea1-a510-652f-7d54-fc9bd7dd54db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2 Address:ca34c9e9-77b8-12d7-8c40-3462bfddffe2} {Suffrage:Voter ID:server-aceceea1-a510-652f-7d54-fc9bd7dd54db Address:aceceea1-a510-652f-7d54-fc9bd7dd54db} {Suffrage:Voter ID:server-16088dbc-1f08-4746-53fb-f9bc85878227 Address:16088dbc-1f08-4746-53fb-f9bc85878227}]"
2025-06-26T05:13:23.190Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2 Address:ca34c9e9-77b8-12d7-8c40-3462bfddffe2} {Suffrage:Voter ID:server-aceceea1-a510-652f-7d54-fc9bd7dd54db Address:aceceea1-a510-652f-7d54-fc9bd7dd54db} {Suffrage:Voter ID:server-16088dbc-1f08-4746-53fb-f9bc85878227 Address:16088dbc-1f08-4746-53fb-f9bc85878227}]"
2025-06-26T05:13:23.190Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:23.190Z [INFO]  server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2: entering follower state: follower="Node at ca34c9e9-77b8-12d7-8c40-3462bfddffe2 [Follower]" leader-address= leader-id=
2025-06-26T05:13:23.190Z [INFO]  server-aceceea1-a510-652f-7d54-fc9bd7dd54db: entering follower state: follower="Node at aceceea1-a510-652f-7d54-fc9bd7dd54db [Follower]" leader-address= leader-id=
2025-06-26T05:13:23.190Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: entering follower state: follower="Node at 16088dbc-1f08-4746-53fb-f9bc85878227 [Follower]" leader-address= leader-id=
2025-06-26T05:13:23.965Z [WARN]  server-16088dbc-1f08-4746-53fb-f9bc85878227: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:23.965Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: entering candidate state: node="Node at 16088dbc-1f08-4746-53fb-f9bc85878227 [Candidate]" term=2
2025-06-26T05:13:23.965Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: election won: term=2 tally=2
2025-06-26T05:13:23.965Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: entering leader state: leader="Node at 16088dbc-1f08-4746-53fb-f9bc85878227 [Leader]"
2025-06-26T05:13:23.965Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: added peer, starting replication: peer=server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2
2025-06-26T05:13:23.965Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: added peer, starting replication: peer=server-aceceea1-a510-652f-7d54-fc9bd7dd54db
2025-06-26T05:13:23.965Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: pipelining replication: peer="{Voter server-aceceea1-a510-652f-7d54-fc9bd7dd54db aceceea1-a510-652f-7d54-fc9bd7dd54db}"
2025-06-26T05:13:23.966Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: pipelining replication: peer="{Voter server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2 ca34c9e9-77b8-12d7-8c40-3462bfddffe2}"
2025-06-26T05:13:24.973Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:23.966065403 +0000 UTC m=+32.115175681 (1 nodes), 775.58388ms from start of poll, 775.77816ms from cluster start. Timeout at 2025-06-26 05:13:24.973567643 +0000 UTC m=+33.122677921, 1.00750224s after stability
2025-06-26T05:13:24.973Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: starting snapshot up to: index=12
2025-06-26T05:13:24.974Z [INFO]  creating new snapshot: path=/tmp/raft3759857341/snapshots/2-12-1750914804974.tmp
2025-06-26T05:13:24.974Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: snapshot complete up to: index=12
2025-06-26T05:13:24.974Z [INFO]  creating new snapshot: path=/tmp/raft3759857341/snapshots/2-23-1750914804974.tmp
2025-06-26T05:13:24.974Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: copied to local snapshot: bytes=71
2025-06-26T05:13:24.975Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: snapshot restore progress: id=2-23-1750914804974 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:24.975Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: removing all old logs from log store
2025-06-26T05:13:24.975Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: compacting logs: from=1 to=22
2025-06-26T05:13:24.975Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: restored user snapshot: index=23
2025-06-26T05:13:24.975Z [ERROR] server-16088dbc-1f08-4746-53fb-f9bc85878227: failed to get log: index=22 error="log not found"
2025-06-26T05:13:24.975Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: aborting pipeline replication: peer="{Voter server-aceceea1-a510-652f-7d54-fc9bd7dd54db aceceea1-a510-652f-7d54-fc9bd7dd54db}"
2025-06-26T05:13:24.975Z [ERROR] server-16088dbc-1f08-4746-53fb-f9bc85878227: failed to get log: index=22 error="log not found"
2025-06-26T05:13:24.975Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: aborting pipeline replication: peer="{Voter server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2 ca34c9e9-77b8-12d7-8c40-3462bfddffe2}"
2025-06-26T05:13:24.981Z [ERROR] server-16088dbc-1f08-4746-53fb-f9bc85878227: failed to get log: index=22 error="log not found"
2025-06-26T05:13:24.981Z [INFO]  creating new snapshot: path=/tmp/raft2164392168/snapshots/2-23-1750914804981.tmp
2025-06-26T05:13:24.981Z [INFO]  server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:24.982Z [INFO]  server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2: copied to local snapshot: bytes=71
2025-06-26T05:13:24.982Z [INFO]  server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2: snapshot restore progress: id=2-23-1750914804981 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:24.982Z [INFO]  server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2: removing all old logs from log store
2025-06-26T05:13:24.982Z [INFO]  server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2: compacting logs: from=1 to=22
2025-06-26T05:13:24.982Z [INFO]  server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2: Installed remote snapshot
2025-06-26T05:13:24.982Z [WARN]  server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2: skipping application of old log: index=22
2025-06-26T05:13:24.982Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: pipelining replication: peer="{Voter server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2 ca34c9e9-77b8-12d7-8c40-3462bfddffe2}"
2025-06-26T05:13:24.982Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:24.985Z [ERROR] server-16088dbc-1f08-4746-53fb-f9bc85878227: failed to get log: index=22 error="log not found"
2025-06-26T05:13:24.985Z [INFO]  creating new snapshot: path=/tmp/raft3281337103/snapshots/2-23-1750914804985.tmp
2025-06-26T05:13:24.985Z [INFO]  server-aceceea1-a510-652f-7d54-fc9bd7dd54db: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:24.986Z [INFO]  server-aceceea1-a510-652f-7d54-fc9bd7dd54db: copied to local snapshot: bytes=71
2025-06-26T05:13:24.986Z [INFO]  server-aceceea1-a510-652f-7d54-fc9bd7dd54db: snapshot restore progress: id=2-23-1750914804985 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:24.986Z [INFO]  server-aceceea1-a510-652f-7d54-fc9bd7dd54db: removing all old logs from log store
2025-06-26T05:13:24.986Z [INFO]  server-aceceea1-a510-652f-7d54-fc9bd7dd54db: compacting logs: from=1 to=22
2025-06-26T05:13:24.986Z [INFO]  server-aceceea1-a510-652f-7d54-fc9bd7dd54db: Installed remote snapshot
2025-06-26T05:13:24.986Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: pipelining replication: peer="{Voter server-aceceea1-a510-652f-7d54-fc9bd7dd54db aceceea1-a510-652f-7d54-fc9bd7dd54db}"
2025-06-26T05:13:25.989Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:24.982247923 +0000 UTC m=+33.131358201 (2 nodes), 640ns from start of poll, 1.79196068s from cluster start. Timeout at 2025-06-26 05:13:25.989570123 +0000 UTC m=+34.138680401, 1.0073222s after stability
2025-06-26T05:13:26.009Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: aborting pipeline replication: peer="{Voter server-aceceea1-a510-652f-7d54-fc9bd7dd54db aceceea1-a510-652f-7d54-fc9bd7dd54db}"
2025-06-26T05:13:26.009Z [INFO]  server-16088dbc-1f08-4746-53fb-f9bc85878227: aborting pipeline replication: peer="{Voter server-ca34c9e9-77b8-12d7-8c40-3462bfddffe2 ca34c9e9-77b8-12d7-8c40-3462bfddffe2}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2025-06-26T05:13:26.010Z [INFO]  server-917ad8d5-ccbb-d537-0c24-f9f345854adb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-917ad8d5-ccbb-d537-0c24-f9f345854adb Address:917ad8d5-ccbb-d537-0c24-f9f345854adb} {Suffrage:Voter ID:server-9fbc9ba9-977b-624a-08e2-9767bd4477c0 Address:9fbc9ba9-977b-624a-08e2-9767bd4477c0} {Suffrage:Voter ID:server-c9540fdb-80dd-9309-07a9-e1c114683237 Address:c9540fdb-80dd-9309-07a9-e1c114683237}]"
2025-06-26T05:13:26.010Z [INFO]  server-9fbc9ba9-977b-624a-08e2-9767bd4477c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-917ad8d5-ccbb-d537-0c24-f9f345854adb Address:917ad8d5-ccbb-d537-0c24-f9f345854adb} {Suffrage:Voter ID:server-9fbc9ba9-977b-624a-08e2-9767bd4477c0 Address:9fbc9ba9-977b-624a-08e2-9767bd4477c0} {Suffrage:Voter ID:server-c9540fdb-80dd-9309-07a9-e1c114683237 Address:c9540fdb-80dd-9309-07a9-e1c114683237}]"
2025-06-26T05:13:26.010Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-917ad8d5-ccbb-d537-0c24-f9f345854adb Address:917ad8d5-ccbb-d537-0c24-f9f345854adb} {Suffrage:Voter ID:server-9fbc9ba9-977b-624a-08e2-9767bd4477c0 Address:9fbc9ba9-977b-624a-08e2-9767bd4477c0} {Suffrage:Voter ID:server-c9540fdb-80dd-9309-07a9-e1c114683237 Address:c9540fdb-80dd-9309-07a9-e1c114683237}]"
2025-06-26T05:13:26.010Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:26.010Z [INFO]  server-917ad8d5-ccbb-d537-0c24-f9f345854adb: entering follower state: follower="Node at 917ad8d5-ccbb-d537-0c24-f9f345854adb [Follower]" leader-address= leader-id=
2025-06-26T05:13:26.010Z [INFO]  server-9fbc9ba9-977b-624a-08e2-9767bd4477c0: entering follower state: follower="Node at 9fbc9ba9-977b-624a-08e2-9767bd4477c0 [Follower]" leader-address= leader-id=
2025-06-26T05:13:26.010Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: entering follower state: follower="Node at c9540fdb-80dd-9309-07a9-e1c114683237 [Follower]" leader-address= leader-id=
2025-06-26T05:13:26.637Z [WARN]  server-c9540fdb-80dd-9309-07a9-e1c114683237: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:26.637Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: entering candidate state: node="Node at c9540fdb-80dd-9309-07a9-e1c114683237 [Candidate]" term=2
2025-06-26T05:13:26.637Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: election won: term=2 tally=2
2025-06-26T05:13:26.637Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: entering leader state: leader="Node at c9540fdb-80dd-9309-07a9-e1c114683237 [Leader]"
2025-06-26T05:13:26.637Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: added peer, starting replication: peer=server-917ad8d5-ccbb-d537-0c24-f9f345854adb
2025-06-26T05:13:26.637Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: added peer, starting replication: peer=server-9fbc9ba9-977b-624a-08e2-9767bd4477c0
2025-06-26T05:13:26.637Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: pipelining replication: peer="{Voter server-9fbc9ba9-977b-624a-08e2-9767bd4477c0 9fbc9ba9-977b-624a-08e2-9767bd4477c0}"
2025-06-26T05:13:26.637Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: pipelining replication: peer="{Voter server-917ad8d5-ccbb-d537-0c24-f9f345854adb 917ad8d5-ccbb-d537-0c24-f9f345854adb}"
2025-06-26T05:13:27.645Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:26.637935043 +0000 UTC m=+34.787045281 (1 nodes), 627.56892ms from start of poll, 627.76136ms from cluster start. Timeout at 2025-06-26 05:13:27.645636483 +0000 UTC m=+35.794746841, 1.00770156s after stability
2025-06-26T05:13:27.646Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: starting snapshot up to: index=12
2025-06-26T05:13:27.646Z [INFO]  creating new snapshot: path=/tmp/raft3033542692/snapshots/2-12-1750914807646.tmp
2025-06-26T05:13:27.646Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: snapshot complete up to: index=12
2025-06-26T05:13:27.646Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5c47bc8-4d74-387b-0689-07900dafe956 Address:f5c47bc8-4d74-387b-0689-07900dafe956} {Suffrage:Voter ID:server-a9fb94a5-8ed9-1fb8-634e-077e12077d70 Address:a9fb94a5-8ed9-1fb8-634e-077e12077d70} {Suffrage:Voter ID:server-6ec775b9-e92a-4235-4c69-e8a467a8c581 Address:6ec775b9-e92a-4235-4c69-e8a467a8c581}]"
2025-06-26T05:13:27.647Z [INFO]  server-a9fb94a5-8ed9-1fb8-634e-077e12077d70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5c47bc8-4d74-387b-0689-07900dafe956 Address:f5c47bc8-4d74-387b-0689-07900dafe956} {Suffrage:Voter ID:server-a9fb94a5-8ed9-1fb8-634e-077e12077d70 Address:a9fb94a5-8ed9-1fb8-634e-077e12077d70} {Suffrage:Voter ID:server-6ec775b9-e92a-4235-4c69-e8a467a8c581 Address:6ec775b9-e92a-4235-4c69-e8a467a8c581}]"
2025-06-26T05:13:27.647Z [INFO]  server-6ec775b9-e92a-4235-4c69-e8a467a8c581: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5c47bc8-4d74-387b-0689-07900dafe956 Address:f5c47bc8-4d74-387b-0689-07900dafe956} {Suffrage:Voter ID:server-a9fb94a5-8ed9-1fb8-634e-077e12077d70 Address:a9fb94a5-8ed9-1fb8-634e-077e12077d70} {Suffrage:Voter ID:server-6ec775b9-e92a-4235-4c69-e8a467a8c581 Address:6ec775b9-e92a-4235-4c69-e8a467a8c581}]"
2025-06-26T05:13:27.647Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:27.647Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: entering follower state: follower="Node at f5c47bc8-4d74-387b-0689-07900dafe956 [Follower]" leader-address= leader-id=
2025-06-26T05:13:27.647Z [INFO]  server-a9fb94a5-8ed9-1fb8-634e-077e12077d70: entering follower state: follower="Node at a9fb94a5-8ed9-1fb8-634e-077e12077d70 [Follower]" leader-address= leader-id=
2025-06-26T05:13:27.647Z [INFO]  server-6ec775b9-e92a-4235-4c69-e8a467a8c581: entering follower state: follower="Node at 6ec775b9-e92a-4235-4c69-e8a467a8c581 [Follower]" leader-address= leader-id=
2025-06-26T05:13:28.197Z [WARN]  server-f5c47bc8-4d74-387b-0689-07900dafe956: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:28.197Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: entering candidate state: node="Node at f5c47bc8-4d74-387b-0689-07900dafe956 [Candidate]" term=2
2025-06-26T05:13:28.197Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: election won: term=2 tally=2
2025-06-26T05:13:28.197Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: entering leader state: leader="Node at f5c47bc8-4d74-387b-0689-07900dafe956 [Leader]"
2025-06-26T05:13:28.197Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: added peer, starting replication: peer=server-a9fb94a5-8ed9-1fb8-634e-077e12077d70
2025-06-26T05:13:28.197Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: added peer, starting replication: peer=server-6ec775b9-e92a-4235-4c69-e8a467a8c581
2025-06-26T05:13:28.197Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: pipelining replication: peer="{Voter server-6ec775b9-e92a-4235-4c69-e8a467a8c581 6ec775b9-e92a-4235-4c69-e8a467a8c581}"
2025-06-26T05:13:28.197Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: pipelining replication: peer="{Voter server-a9fb94a5-8ed9-1fb8-634e-077e12077d70 a9fb94a5-8ed9-1fb8-634e-077e12077d70}"
2025-06-26T05:13:29.205Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:28.197788723 +0000 UTC m=+36.346899001 (1 nodes), 550.67364ms from start of poll, 550.8728ms from cluster start. Timeout at 2025-06-26 05:13:29.205654083 +0000 UTC m=+37.354764321, 1.00786532s after stability
2025-06-26T05:13:29.205Z [INFO]  creating new snapshot: path=/tmp/raft3225779080/snapshots/2-14-1750914809205.tmp
2025-06-26T05:13:29.206Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: copied to local snapshot: bytes=71
2025-06-26T05:13:29.206Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: snapshot restore progress: id=2-14-1750914809205 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:29.206Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: restored user snapshot: index=14
2025-06-26T05:13:29.206Z [ERROR] server-f5c47bc8-4d74-387b-0689-07900dafe956: failed to get log: index=3 error="log not found"
2025-06-26T05:13:29.206Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: aborting pipeline replication: peer="{Voter server-6ec775b9-e92a-4235-4c69-e8a467a8c581 6ec775b9-e92a-4235-4c69-e8a467a8c581}"
2025-06-26T05:13:29.206Z [ERROR] server-f5c47bc8-4d74-387b-0689-07900dafe956: failed to get log: index=3 error="log not found"
2025-06-26T05:13:29.206Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: aborting pipeline replication: peer="{Voter server-a9fb94a5-8ed9-1fb8-634e-077e12077d70 a9fb94a5-8ed9-1fb8-634e-077e12077d70}"
2025-06-26T05:13:29.217Z [ERROR] server-f5c47bc8-4d74-387b-0689-07900dafe956: failed to get log: index=3 error="log not found"
2025-06-26T05:13:29.217Z [INFO]  creating new snapshot: path=/tmp/raft2546077767/snapshots/2-14-1750914809217.tmp
2025-06-26T05:13:29.217Z [INFO]  server-a9fb94a5-8ed9-1fb8-634e-077e12077d70: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:29.218Z [INFO]  server-a9fb94a5-8ed9-1fb8-634e-077e12077d70: copied to local snapshot: bytes=71
2025-06-26T05:13:29.218Z [INFO]  server-a9fb94a5-8ed9-1fb8-634e-077e12077d70: snapshot restore progress: id=2-14-1750914809217 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:29.218Z [INFO]  server-a9fb94a5-8ed9-1fb8-634e-077e12077d70: Installed remote snapshot
2025-06-26T05:13:29.218Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: pipelining replication: peer="{Voter server-a9fb94a5-8ed9-1fb8-634e-077e12077d70 a9fb94a5-8ed9-1fb8-634e-077e12077d70}"
2025-06-26T05:13:29.218Z [ERROR] server-f5c47bc8-4d74-387b-0689-07900dafe956: failed to get log: index=3 error="log not found"
2025-06-26T05:13:29.218Z [INFO]  creating new snapshot: path=/tmp/raft3313615555/snapshots/2-14-1750914809218.tmp
2025-06-26T05:13:29.218Z [INFO]  server-6ec775b9-e92a-4235-4c69-e8a467a8c581: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:29.218Z [INFO]  server-6ec775b9-e92a-4235-4c69-e8a467a8c581: copied to local snapshot: bytes=71
2025-06-26T05:13:29.218Z [INFO]  server-6ec775b9-e92a-4235-4c69-e8a467a8c581: snapshot restore progress: id=2-14-1750914809218 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:29.218Z [INFO]  server-6ec775b9-e92a-4235-4c69-e8a467a8c581: Installed remote snapshot
2025-06-26T05:13:29.218Z [INFO]  server-f5c47bc8-4d74-387b-0689-07900dafe956: pipelining replication: peer="{Voter server-6ec775b9-e92a-4235-4c69-e8a467a8c581 6ec775b9-e92a-4235-4c69-e8a467a8c581}"
2025-06-26T05:13:29.218Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:30.225Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:29.218739403 +0000 UTC m=+37.367849681 (2 nodes), 440ns from start of poll, 1.57182348s from cluster start. Timeout at 2025-06-26 05:13:30.225604363 +0000 UTC m=+38.374714641, 1.00686496s after stability
2025-06-26T05:13:30.249Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: aborting pipeline replication: peer="{Voter server-9fbc9ba9-977b-624a-08e2-9767bd4477c0 9fbc9ba9-977b-624a-08e2-9767bd4477c0}"
2025-06-26T05:13:30.249Z [INFO]  server-c9540fdb-80dd-9309-07a9-e1c114683237: aborting pipeline replication: peer="{Voter server-917ad8d5-ccbb-d537-0c24-f9f345854adb 917ad8d5-ccbb-d537-0c24-f9f345854adb}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2025-06-26T05:13:30.250Z [INFO]  server-de809a7f-7fed-89e7-7657-20653ddf5e9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de809a7f-7fed-89e7-7657-20653ddf5e9f Address:de809a7f-7fed-89e7-7657-20653ddf5e9f} {Suffrage:Voter ID:server-b224de3c-fb75-c2d0-d94a-775af0e9c885 Address:b224de3c-fb75-c2d0-d94a-775af0e9c885} {Suffrage:Voter ID:server-d7ac447b-23ce-9fe1-2f9a-be7da2406212 Address:d7ac447b-23ce-9fe1-2f9a-be7da2406212}]"
2025-06-26T05:13:30.250Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de809a7f-7fed-89e7-7657-20653ddf5e9f Address:de809a7f-7fed-89e7-7657-20653ddf5e9f} {Suffrage:Voter ID:server-b224de3c-fb75-c2d0-d94a-775af0e9c885 Address:b224de3c-fb75-c2d0-d94a-775af0e9c885} {Suffrage:Voter ID:server-d7ac447b-23ce-9fe1-2f9a-be7da2406212 Address:d7ac447b-23ce-9fe1-2f9a-be7da2406212}]"
2025-06-26T05:13:30.250Z [INFO]  server-d7ac447b-23ce-9fe1-2f9a-be7da2406212: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de809a7f-7fed-89e7-7657-20653ddf5e9f Address:de809a7f-7fed-89e7-7657-20653ddf5e9f} {Suffrage:Voter ID:server-b224de3c-fb75-c2d0-d94a-775af0e9c885 Address:b224de3c-fb75-c2d0-d94a-775af0e9c885} {Suffrage:Voter ID:server-d7ac447b-23ce-9fe1-2f9a-be7da2406212 Address:d7ac447b-23ce-9fe1-2f9a-be7da2406212}]"
2025-06-26T05:13:30.250Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:30.250Z [INFO]  server-de809a7f-7fed-89e7-7657-20653ddf5e9f: entering follower state: follower="Node at de809a7f-7fed-89e7-7657-20653ddf5e9f [Follower]" leader-address= leader-id=
2025-06-26T05:13:30.250Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: entering follower state: follower="Node at b224de3c-fb75-c2d0-d94a-775af0e9c885 [Follower]" leader-address= leader-id=
2025-06-26T05:13:30.250Z [INFO]  server-d7ac447b-23ce-9fe1-2f9a-be7da2406212: entering follower state: follower="Node at d7ac447b-23ce-9fe1-2f9a-be7da2406212 [Follower]" leader-address= leader-id=
2025-06-26T05:13:30.825Z [WARN]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:30.825Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: entering candidate state: node="Node at b224de3c-fb75-c2d0-d94a-775af0e9c885 [Candidate]" term=2
2025-06-26T05:13:30.825Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: election won: term=2 tally=2
2025-06-26T05:13:30.825Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: entering leader state: leader="Node at b224de3c-fb75-c2d0-d94a-775af0e9c885 [Leader]"
2025-06-26T05:13:30.825Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: added peer, starting replication: peer=server-de809a7f-7fed-89e7-7657-20653ddf5e9f
2025-06-26T05:13:30.825Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: added peer, starting replication: peer=server-d7ac447b-23ce-9fe1-2f9a-be7da2406212
2025-06-26T05:13:30.825Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: pipelining replication: peer="{Voter server-d7ac447b-23ce-9fe1-2f9a-be7da2406212 d7ac447b-23ce-9fe1-2f9a-be7da2406212}"
2025-06-26T05:13:30.826Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: pipelining replication: peer="{Voter server-de809a7f-7fed-89e7-7657-20653ddf5e9f de809a7f-7fed-89e7-7657-20653ddf5e9f}"
2025-06-26T05:13:31.833Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:30.826083243 +0000 UTC m=+38.975193521 (1 nodes), 575.71348ms from start of poll, 575.89784ms from cluster start. Timeout at 2025-06-26 05:13:31.833597723 +0000 UTC m=+39.982708001, 1.00751448s after stability
2025-06-26T05:13:31.834Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: starting snapshot up to: index=12
2025-06-26T05:13:31.834Z [INFO]  creating new snapshot: path=/tmp/raft1856750099/snapshots/2-12-1750914811834.tmp
2025-06-26T05:13:31.834Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: snapshot complete up to: index=12
2025-06-26T05:13:31.835Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd271f83-89bf-1157-cbb8-43454f8db9d7 Address:bd271f83-89bf-1157-cbb8-43454f8db9d7} {Suffrage:Voter ID:server-36c11469-2926-c438-ec51-cfa2a73d65d5 Address:36c11469-2926-c438-ec51-cfa2a73d65d5} {Suffrage:Voter ID:server-8b39098e-d7bf-4748-67bf-bf36ff90a827 Address:8b39098e-d7bf-4748-67bf-bf36ff90a827}]"
2025-06-26T05:13:31.835Z [INFO]  server-36c11469-2926-c438-ec51-cfa2a73d65d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd271f83-89bf-1157-cbb8-43454f8db9d7 Address:bd271f83-89bf-1157-cbb8-43454f8db9d7} {Suffrage:Voter ID:server-36c11469-2926-c438-ec51-cfa2a73d65d5 Address:36c11469-2926-c438-ec51-cfa2a73d65d5} {Suffrage:Voter ID:server-8b39098e-d7bf-4748-67bf-bf36ff90a827 Address:8b39098e-d7bf-4748-67bf-bf36ff90a827}]"
2025-06-26T05:13:31.835Z [INFO]  server-8b39098e-d7bf-4748-67bf-bf36ff90a827: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd271f83-89bf-1157-cbb8-43454f8db9d7 Address:bd271f83-89bf-1157-cbb8-43454f8db9d7} {Suffrage:Voter ID:server-36c11469-2926-c438-ec51-cfa2a73d65d5 Address:36c11469-2926-c438-ec51-cfa2a73d65d5} {Suffrage:Voter ID:server-8b39098e-d7bf-4748-67bf-bf36ff90a827 Address:8b39098e-d7bf-4748-67bf-bf36ff90a827}]"
2025-06-26T05:13:31.835Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:31.835Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: entering follower state: follower="Node at bd271f83-89bf-1157-cbb8-43454f8db9d7 [Follower]" leader-address= leader-id=
2025-06-26T05:13:31.835Z [INFO]  server-36c11469-2926-c438-ec51-cfa2a73d65d5: entering follower state: follower="Node at 36c11469-2926-c438-ec51-cfa2a73d65d5 [Follower]" leader-address= leader-id=
2025-06-26T05:13:31.835Z [INFO]  server-8b39098e-d7bf-4748-67bf-bf36ff90a827: entering follower state: follower="Node at 8b39098e-d7bf-4748-67bf-bf36ff90a827 [Follower]" leader-address= leader-id=
2025-06-26T05:13:32.421Z [WARN]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:32.421Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: entering candidate state: node="Node at bd271f83-89bf-1157-cbb8-43454f8db9d7 [Candidate]" term=2
2025-06-26T05:13:32.421Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: election won: term=2 tally=2
2025-06-26T05:13:32.421Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: entering leader state: leader="Node at bd271f83-89bf-1157-cbb8-43454f8db9d7 [Leader]"
2025-06-26T05:13:32.421Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: added peer, starting replication: peer=server-36c11469-2926-c438-ec51-cfa2a73d65d5
2025-06-26T05:13:32.421Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: added peer, starting replication: peer=server-8b39098e-d7bf-4748-67bf-bf36ff90a827
2025-06-26T05:13:32.421Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: pipelining replication: peer="{Voter server-8b39098e-d7bf-4748-67bf-bf36ff90a827 8b39098e-d7bf-4748-67bf-bf36ff90a827}"
2025-06-26T05:13:32.422Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: pipelining replication: peer="{Voter server-36c11469-2926-c438-ec51-cfa2a73d65d5 36c11469-2926-c438-ec51-cfa2a73d65d5}"
2025-06-26T05:13:33.429Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:32.422129403 +0000 UTC m=+40.571239681 (1 nodes), 586.95996ms from start of poll, 587.15324ms from cluster start. Timeout at 2025-06-26 05:13:33.429942883 +0000 UTC m=+41.579053121, 1.00781344s after stability
2025-06-26T05:13:33.430Z [INFO]  creating new snapshot: path=/tmp/raft1897499428/snapshots/2-14-1750914813430.tmp
2025-06-26T05:13:33.430Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: copied to local snapshot: bytes=71
2025-06-26T05:13:33.430Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: snapshot restore progress: id=2-14-1750914813430 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:33.430Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: removing all old logs from log store
2025-06-26T05:13:33.430Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: compacting logs: from=1 to=2
2025-06-26T05:13:33.430Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: restored user snapshot: index=14
2025-06-26T05:13:33.430Z [ERROR] server-bd271f83-89bf-1157-cbb8-43454f8db9d7: failed to get log: index=2 error="log not found"
2025-06-26T05:13:33.430Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: aborting pipeline replication: peer="{Voter server-8b39098e-d7bf-4748-67bf-bf36ff90a827 8b39098e-d7bf-4748-67bf-bf36ff90a827}"
2025-06-26T05:13:33.430Z [ERROR] server-bd271f83-89bf-1157-cbb8-43454f8db9d7: failed to get log: index=2 error="log not found"
2025-06-26T05:13:33.430Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: aborting pipeline replication: peer="{Voter server-36c11469-2926-c438-ec51-cfa2a73d65d5 36c11469-2926-c438-ec51-cfa2a73d65d5}"
2025-06-26T05:13:33.441Z [ERROR] server-bd271f83-89bf-1157-cbb8-43454f8db9d7: failed to get log: index=2 error="log not found"
2025-06-26T05:13:33.442Z [INFO]  creating new snapshot: path=/tmp/raft3381579055/snapshots/2-14-1750914813442.tmp
2025-06-26T05:13:33.442Z [INFO]  server-8b39098e-d7bf-4748-67bf-bf36ff90a827: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:33.442Z [INFO]  server-8b39098e-d7bf-4748-67bf-bf36ff90a827: copied to local snapshot: bytes=71
2025-06-26T05:13:33.442Z [INFO]  server-8b39098e-d7bf-4748-67bf-bf36ff90a827: snapshot restore progress: id=2-14-1750914813442 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:33.442Z [INFO]  server-8b39098e-d7bf-4748-67bf-bf36ff90a827: removing all old logs from log store
2025-06-26T05:13:33.442Z [INFO]  server-8b39098e-d7bf-4748-67bf-bf36ff90a827: compacting logs: from=1 to=2
2025-06-26T05:13:33.442Z [INFO]  server-8b39098e-d7bf-4748-67bf-bf36ff90a827: Installed remote snapshot
2025-06-26T05:13:33.442Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: pipelining replication: peer="{Voter server-8b39098e-d7bf-4748-67bf-bf36ff90a827 8b39098e-d7bf-4748-67bf-bf36ff90a827}"
2025-06-26T05:13:33.442Z [ERROR] server-bd271f83-89bf-1157-cbb8-43454f8db9d7: failed to get log: index=2 error="log not found"
2025-06-26T05:13:33.442Z [INFO]  creating new snapshot: path=/tmp/raft1655615954/snapshots/2-14-1750914813442.tmp
2025-06-26T05:13:33.442Z [INFO]  server-36c11469-2926-c438-ec51-cfa2a73d65d5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:33.442Z [INFO]  server-36c11469-2926-c438-ec51-cfa2a73d65d5: copied to local snapshot: bytes=71
2025-06-26T05:13:33.442Z [INFO]  server-36c11469-2926-c438-ec51-cfa2a73d65d5: snapshot restore progress: id=2-14-1750914813442 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:33.442Z [INFO]  server-36c11469-2926-c438-ec51-cfa2a73d65d5: removing all old logs from log store
2025-06-26T05:13:33.442Z [INFO]  server-36c11469-2926-c438-ec51-cfa2a73d65d5: compacting logs: from=1 to=2
2025-06-26T05:13:33.442Z [INFO]  server-36c11469-2926-c438-ec51-cfa2a73d65d5: Installed remote snapshot
2025-06-26T05:13:33.442Z [INFO]  server-bd271f83-89bf-1157-cbb8-43454f8db9d7: pipelining replication: peer="{Voter server-36c11469-2926-c438-ec51-cfa2a73d65d5 36c11469-2926-c438-ec51-cfa2a73d65d5}"
2025-06-26T05:13:33.442Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:34.449Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:33.442988683 +0000 UTC m=+41.592098961 (2 nodes), 440ns from start of poll, 1.60801252s from cluster start. Timeout at 2025-06-26 05:13:34.449614203 +0000 UTC m=+42.598724481, 1.00662552s after stability
2025-06-26T05:13:34.465Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: aborting pipeline replication: peer="{Voter server-d7ac447b-23ce-9fe1-2f9a-be7da2406212 d7ac447b-23ce-9fe1-2f9a-be7da2406212}"
2025-06-26T05:13:34.465Z [INFO]  server-b224de3c-fb75-c2d0-d94a-775af0e9c885: aborting pipeline replication: peer="{Voter server-de809a7f-7fed-89e7-7657-20653ddf5e9f de809a7f-7fed-89e7-7657-20653ddf5e9f}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2025-06-26T05:13:34.466Z [INFO]  server-4ed5ca79-5b77-2b53-a970-8426a5c53a72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ed5ca79-5b77-2b53-a970-8426a5c53a72 Address:4ed5ca79-5b77-2b53-a970-8426a5c53a72} {Suffrage:Voter ID:server-9251a923-869e-5bae-ed37-a376d30cb7b5 Address:9251a923-869e-5bae-ed37-a376d30cb7b5} {Suffrage:Voter ID:server-affe80ee-e423-5b51-43ee-f661bb220c12 Address:affe80ee-e423-5b51-43ee-f661bb220c12}]"
2025-06-26T05:13:34.466Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ed5ca79-5b77-2b53-a970-8426a5c53a72 Address:4ed5ca79-5b77-2b53-a970-8426a5c53a72} {Suffrage:Voter ID:server-9251a923-869e-5bae-ed37-a376d30cb7b5 Address:9251a923-869e-5bae-ed37-a376d30cb7b5} {Suffrage:Voter ID:server-affe80ee-e423-5b51-43ee-f661bb220c12 Address:affe80ee-e423-5b51-43ee-f661bb220c12}]"
2025-06-26T05:13:34.466Z [INFO]  server-affe80ee-e423-5b51-43ee-f661bb220c12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ed5ca79-5b77-2b53-a970-8426a5c53a72 Address:4ed5ca79-5b77-2b53-a970-8426a5c53a72} {Suffrage:Voter ID:server-9251a923-869e-5bae-ed37-a376d30cb7b5 Address:9251a923-869e-5bae-ed37-a376d30cb7b5} {Suffrage:Voter ID:server-affe80ee-e423-5b51-43ee-f661bb220c12 Address:affe80ee-e423-5b51-43ee-f661bb220c12}]"
2025-06-26T05:13:34.466Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:34.466Z [INFO]  server-4ed5ca79-5b77-2b53-a970-8426a5c53a72: entering follower state: follower="Node at 4ed5ca79-5b77-2b53-a970-8426a5c53a72 [Follower]" leader-address= leader-id=
2025-06-26T05:13:34.466Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: entering follower state: follower="Node at 9251a923-869e-5bae-ed37-a376d30cb7b5 [Follower]" leader-address= leader-id=
2025-06-26T05:13:34.466Z [INFO]  server-affe80ee-e423-5b51-43ee-f661bb220c12: entering follower state: follower="Node at affe80ee-e423-5b51-43ee-f661bb220c12 [Follower]" leader-address= leader-id=
2025-06-26T05:13:35.001Z [WARN]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:35.001Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: entering candidate state: node="Node at 9251a923-869e-5bae-ed37-a376d30cb7b5 [Candidate]" term=2
2025-06-26T05:13:35.001Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: election won: term=2 tally=2
2025-06-26T05:13:35.001Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: entering leader state: leader="Node at 9251a923-869e-5bae-ed37-a376d30cb7b5 [Leader]"
2025-06-26T05:13:35.001Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: added peer, starting replication: peer=server-4ed5ca79-5b77-2b53-a970-8426a5c53a72
2025-06-26T05:13:35.001Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: added peer, starting replication: peer=server-affe80ee-e423-5b51-43ee-f661bb220c12
2025-06-26T05:13:35.001Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: pipelining replication: peer="{Voter server-affe80ee-e423-5b51-43ee-f661bb220c12 affe80ee-e423-5b51-43ee-f661bb220c12}"
2025-06-26T05:13:35.001Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: pipelining replication: peer="{Voter server-4ed5ca79-5b77-2b53-a970-8426a5c53a72 4ed5ca79-5b77-2b53-a970-8426a5c53a72}"
2025-06-26T05:13:36.009Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:35.001960283 +0000 UTC m=+43.151070561 (1 nodes), 535.54172ms from start of poll, 535.72396ms from cluster start. Timeout at 2025-06-26 05:13:36.009855003 +0000 UTC m=+44.158965281, 1.00789472s after stability
2025-06-26T05:13:36.010Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: starting snapshot up to: index=12
2025-06-26T05:13:36.010Z [INFO]  creating new snapshot: path=/tmp/raft1667802024/snapshots/2-12-1750914816010.tmp
2025-06-26T05:13:36.010Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: snapshot complete up to: index=12
2025-06-26T05:13:36.011Z [INFO]  creating new snapshot: path=/tmp/raft1667802024/snapshots/2-23-1750914816011.tmp
2025-06-26T05:13:36.011Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: copied to local snapshot: bytes=71
2025-06-26T05:13:36.011Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: snapshot restore progress: id=2-23-1750914816011 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:36.011Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: restored user snapshot: index=23
2025-06-26T05:13:36.011Z [ERROR] server-9251a923-869e-5bae-ed37-a376d30cb7b5: failed to get log: index=23 error="log not found"
2025-06-26T05:13:36.011Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: aborting pipeline replication: peer="{Voter server-affe80ee-e423-5b51-43ee-f661bb220c12 affe80ee-e423-5b51-43ee-f661bb220c12}"
2025-06-26T05:13:36.011Z [ERROR] server-9251a923-869e-5bae-ed37-a376d30cb7b5: failed to get log: index=23 error="log not found"
2025-06-26T05:13:36.011Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: aborting pipeline replication: peer="{Voter server-4ed5ca79-5b77-2b53-a970-8426a5c53a72 4ed5ca79-5b77-2b53-a970-8426a5c53a72}"
2025-06-26T05:13:36.011Z [ERROR] server-9251a923-869e-5bae-ed37-a376d30cb7b5: failed to get log: index=23 error="log not found"
2025-06-26T05:13:36.011Z [INFO]  creating new snapshot: path=/tmp/raft1014289795/snapshots/2-23-1750914816011.tmp
2025-06-26T05:13:36.011Z [INFO]  server-4ed5ca79-5b77-2b53-a970-8426a5c53a72: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:36.011Z [INFO]  server-4ed5ca79-5b77-2b53-a970-8426a5c53a72: copied to local snapshot: bytes=71
2025-06-26T05:13:36.011Z [INFO]  server-4ed5ca79-5b77-2b53-a970-8426a5c53a72: snapshot restore progress: id=2-23-1750914816011 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:36.011Z [INFO]  server-4ed5ca79-5b77-2b53-a970-8426a5c53a72: Installed remote snapshot
2025-06-26T05:13:36.011Z [WARN]  server-4ed5ca79-5b77-2b53-a970-8426a5c53a72: skipping application of old log: index=22
2025-06-26T05:13:36.011Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: pipelining replication: peer="{Voter server-4ed5ca79-5b77-2b53-a970-8426a5c53a72 4ed5ca79-5b77-2b53-a970-8426a5c53a72}"
2025-06-26T05:13:36.011Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:36.021Z [ERROR] server-9251a923-869e-5bae-ed37-a376d30cb7b5: failed to get log: index=23 error="log not found"
2025-06-26T05:13:36.022Z [INFO]  creating new snapshot: path=/tmp/raft2283918732/snapshots/2-23-1750914816022.tmp
2025-06-26T05:13:36.022Z [INFO]  server-affe80ee-e423-5b51-43ee-f661bb220c12: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:36.022Z [INFO]  server-affe80ee-e423-5b51-43ee-f661bb220c12: copied to local snapshot: bytes=71
2025-06-26T05:13:36.022Z [INFO]  server-affe80ee-e423-5b51-43ee-f661bb220c12: snapshot restore progress: id=2-23-1750914816022 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:36.022Z [INFO]  server-affe80ee-e423-5b51-43ee-f661bb220c12: Installed remote snapshot
2025-06-26T05:13:36.022Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: pipelining replication: peer="{Voter server-affe80ee-e423-5b51-43ee-f661bb220c12 affe80ee-e423-5b51-43ee-f661bb220c12}"
2025-06-26T05:13:37.017Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:36.011912203 +0000 UTC m=+44.161022481 (2 nodes), 480ns from start of poll, 1.54567588s from cluster start. Timeout at 2025-06-26 05:13:37.017859563 +0000 UTC m=+45.166969841, 1.00594736s after stability
2025-06-26T05:13:37.033Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: aborting pipeline replication: peer="{Voter server-affe80ee-e423-5b51-43ee-f661bb220c12 affe80ee-e423-5b51-43ee-f661bb220c12}"
2025-06-26T05:13:37.033Z [INFO]  server-9251a923-869e-5bae-ed37-a376d30cb7b5: aborting pipeline replication: peer="{Voter server-4ed5ca79-5b77-2b53-a970-8426a5c53a72 4ed5ca79-5b77-2b53-a970-8426a5c53a72}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2025-06-26T05:13:37.034Z [INFO]  server-4b911690-8768-5a1c-9901-f1cb3397b1fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b911690-8768-5a1c-9901-f1cb3397b1fa Address:4b911690-8768-5a1c-9901-f1cb3397b1fa} {Suffrage:Voter ID:server-04c79682-b6d6-3a28-a0e9-538399e8b148 Address:04c79682-b6d6-3a28-a0e9-538399e8b148} {Suffrage:Voter ID:server-5606b784-daf3-94d5-64dd-fea909b7ce34 Address:5606b784-daf3-94d5-64dd-fea909b7ce34}]"
2025-06-26T05:13:37.034Z [INFO]  server-04c79682-b6d6-3a28-a0e9-538399e8b148: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b911690-8768-5a1c-9901-f1cb3397b1fa Address:4b911690-8768-5a1c-9901-f1cb3397b1fa} {Suffrage:Voter ID:server-04c79682-b6d6-3a28-a0e9-538399e8b148 Address:04c79682-b6d6-3a28-a0e9-538399e8b148} {Suffrage:Voter ID:server-5606b784-daf3-94d5-64dd-fea909b7ce34 Address:5606b784-daf3-94d5-64dd-fea909b7ce34}]"
2025-06-26T05:13:37.034Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b911690-8768-5a1c-9901-f1cb3397b1fa Address:4b911690-8768-5a1c-9901-f1cb3397b1fa} {Suffrage:Voter ID:server-04c79682-b6d6-3a28-a0e9-538399e8b148 Address:04c79682-b6d6-3a28-a0e9-538399e8b148} {Suffrage:Voter ID:server-5606b784-daf3-94d5-64dd-fea909b7ce34 Address:5606b784-daf3-94d5-64dd-fea909b7ce34}]"
2025-06-26T05:13:37.034Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:37.034Z [INFO]  server-4b911690-8768-5a1c-9901-f1cb3397b1fa: entering follower state: follower="Node at 4b911690-8768-5a1c-9901-f1cb3397b1fa [Follower]" leader-address= leader-id=
2025-06-26T05:13:37.034Z [INFO]  server-04c79682-b6d6-3a28-a0e9-538399e8b148: entering follower state: follower="Node at 04c79682-b6d6-3a28-a0e9-538399e8b148 [Follower]" leader-address= leader-id=
2025-06-26T05:13:37.034Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: entering follower state: follower="Node at 5606b784-daf3-94d5-64dd-fea909b7ce34 [Follower]" leader-address= leader-id=
2025-06-26T05:13:37.633Z [WARN]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:37.633Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: entering candidate state: node="Node at 5606b784-daf3-94d5-64dd-fea909b7ce34 [Candidate]" term=2
2025-06-26T05:13:37.633Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: election won: term=2 tally=2
2025-06-26T05:13:37.633Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: entering leader state: leader="Node at 5606b784-daf3-94d5-64dd-fea909b7ce34 [Leader]"
2025-06-26T05:13:37.633Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: added peer, starting replication: peer=server-4b911690-8768-5a1c-9901-f1cb3397b1fa
2025-06-26T05:13:37.633Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: added peer, starting replication: peer=server-04c79682-b6d6-3a28-a0e9-538399e8b148
2025-06-26T05:13:37.633Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: pipelining replication: peer="{Voter server-04c79682-b6d6-3a28-a0e9-538399e8b148 04c79682-b6d6-3a28-a0e9-538399e8b148}"
2025-06-26T05:13:37.633Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: pipelining replication: peer="{Voter server-4b911690-8768-5a1c-9901-f1cb3397b1fa 4b911690-8768-5a1c-9901-f1cb3397b1fa}"
2025-06-26T05:13:38.641Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:37.633995803 +0000 UTC m=+45.783106081 (1 nodes), 599.54456ms from start of poll, 599.7332ms from cluster start. Timeout at 2025-06-26 05:13:38.641606763 +0000 UTC m=+46.790717041, 1.00761096s after stability
2025-06-26T05:13:38.642Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: starting snapshot up to: index=12
2025-06-26T05:13:38.642Z [INFO]  creating new snapshot: path=/tmp/raft4150856616/snapshots/2-12-1750914818642.tmp
2025-06-26T05:13:38.642Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: snapshot complete up to: index=12
2025-06-26T05:13:38.642Z [INFO]  creating new snapshot: path=/tmp/raft4150856616/snapshots/2-23-1750914818642.tmp
2025-06-26T05:13:38.643Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: copied to local snapshot: bytes=71
2025-06-26T05:13:38.643Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: snapshot restore progress: id=2-23-1750914818642 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:38.643Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: removing all old logs from log store
2025-06-26T05:13:38.643Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: compacting logs: from=1 to=22
2025-06-26T05:13:38.643Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: restored user snapshot: index=23
2025-06-26T05:13:38.643Z [ERROR] server-5606b784-daf3-94d5-64dd-fea909b7ce34: failed to get log: index=22 error="log not found"
2025-06-26T05:13:38.643Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: aborting pipeline replication: peer="{Voter server-04c79682-b6d6-3a28-a0e9-538399e8b148 04c79682-b6d6-3a28-a0e9-538399e8b148}"
2025-06-26T05:13:38.643Z [ERROR] server-5606b784-daf3-94d5-64dd-fea909b7ce34: failed to get log: index=22 error="log not found"
2025-06-26T05:13:38.643Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: aborting pipeline replication: peer="{Voter server-4b911690-8768-5a1c-9901-f1cb3397b1fa 4b911690-8768-5a1c-9901-f1cb3397b1fa}"
2025-06-26T05:13:38.653Z [ERROR] server-5606b784-daf3-94d5-64dd-fea909b7ce34: failed to get log: index=22 error="log not found"
2025-06-26T05:13:38.653Z [ERROR] server-5606b784-daf3-94d5-64dd-fea909b7ce34: failed to get log: index=22 error="log not found"
2025-06-26T05:13:38.653Z [INFO]  creating new snapshot: path=/tmp/raft2176989350/snapshots/2-23-1750914818653.tmp
2025-06-26T05:13:38.654Z [INFO]  server-04c79682-b6d6-3a28-a0e9-538399e8b148: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:38.653Z [INFO]  creating new snapshot: path=/tmp/raft2215349731/snapshots/2-23-1750914818653.tmp
2025-06-26T05:13:38.654Z [INFO]  server-04c79682-b6d6-3a28-a0e9-538399e8b148: copied to local snapshot: bytes=71
2025-06-26T05:13:38.654Z [INFO]  server-4b911690-8768-5a1c-9901-f1cb3397b1fa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:38.654Z [INFO]  server-04c79682-b6d6-3a28-a0e9-538399e8b148: snapshot restore progress: id=2-23-1750914818653 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:38.654Z [INFO]  server-04c79682-b6d6-3a28-a0e9-538399e8b148: removing all old logs from log store
2025-06-26T05:13:38.654Z [INFO]  server-04c79682-b6d6-3a28-a0e9-538399e8b148: compacting logs: from=1 to=22
2025-06-26T05:13:38.654Z [INFO]  server-04c79682-b6d6-3a28-a0e9-538399e8b148: Installed remote snapshot
2025-06-26T05:13:38.654Z [WARN]  server-04c79682-b6d6-3a28-a0e9-538399e8b148: skipping application of old log: index=22
2025-06-26T05:13:38.654Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: pipelining replication: peer="{Voter server-04c79682-b6d6-3a28-a0e9-538399e8b148 04c79682-b6d6-3a28-a0e9-538399e8b148}"
2025-06-26T05:13:38.654Z [INFO]  server-4b911690-8768-5a1c-9901-f1cb3397b1fa: copied to local snapshot: bytes=71
2025-06-26T05:13:38.654Z [INFO]  server-4b911690-8768-5a1c-9901-f1cb3397b1fa: snapshot restore progress: id=2-23-1750914818653 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:38.654Z [INFO]  server-4b911690-8768-5a1c-9901-f1cb3397b1fa: removing all old logs from log store
2025-06-26T05:13:38.654Z [INFO]  server-4b911690-8768-5a1c-9901-f1cb3397b1fa: compacting logs: from=1 to=22
2025-06-26T05:13:38.654Z [INFO]  server-4b911690-8768-5a1c-9901-f1cb3397b1fa: Installed remote snapshot
2025-06-26T05:13:38.654Z [WARN]  server-4b911690-8768-5a1c-9901-f1cb3397b1fa: skipping application of old log: index=22
2025-06-26T05:13:38.654Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: pipelining replication: peer="{Voter server-4b911690-8768-5a1c-9901-f1cb3397b1fa 4b911690-8768-5a1c-9901-f1cb3397b1fa}"
2025-06-26T05:13:38.654Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:39.661Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:38.654567283 +0000 UTC m=+46.803677561 (2 nodes), 400ns from start of poll, 1.62030468s from cluster start. Timeout at 2025-06-26 05:13:39.661598083 +0000 UTC m=+47.810708561, 1.007031s after stability
2025-06-26T05:13:39.677Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: aborting pipeline replication: peer="{Voter server-4b911690-8768-5a1c-9901-f1cb3397b1fa 4b911690-8768-5a1c-9901-f1cb3397b1fa}"
2025-06-26T05:13:39.677Z [INFO]  server-5606b784-daf3-94d5-64dd-fea909b7ce34: aborting pipeline replication: peer="{Voter server-04c79682-b6d6-3a28-a0e9-538399e8b148 04c79682-b6d6-3a28-a0e9-538399e8b148}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2025-06-26T05:13:39.678Z [INFO]  server-6d2ac5a7-95b5-5825-adef-ac50da6cf1f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d2ac5a7-95b5-5825-adef-ac50da6cf1f9 Address:6d2ac5a7-95b5-5825-adef-ac50da6cf1f9} {Suffrage:Voter ID:server-32a83739-ad35-e18e-11e4-62c19c6406d5 Address:32a83739-ad35-e18e-11e4-62c19c6406d5} {Suffrage:Voter ID:server-5425c6f7-5e8a-fe21-c000-364edceb916e Address:5425c6f7-5e8a-fe21-c000-364edceb916e}]"
2025-06-26T05:13:39.678Z [INFO]  server-32a83739-ad35-e18e-11e4-62c19c6406d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d2ac5a7-95b5-5825-adef-ac50da6cf1f9 Address:6d2ac5a7-95b5-5825-adef-ac50da6cf1f9} {Suffrage:Voter ID:server-32a83739-ad35-e18e-11e4-62c19c6406d5 Address:32a83739-ad35-e18e-11e4-62c19c6406d5} {Suffrage:Voter ID:server-5425c6f7-5e8a-fe21-c000-364edceb916e Address:5425c6f7-5e8a-fe21-c000-364edceb916e}]"
2025-06-26T05:13:39.678Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d2ac5a7-95b5-5825-adef-ac50da6cf1f9 Address:6d2ac5a7-95b5-5825-adef-ac50da6cf1f9} {Suffrage:Voter ID:server-32a83739-ad35-e18e-11e4-62c19c6406d5 Address:32a83739-ad35-e18e-11e4-62c19c6406d5} {Suffrage:Voter ID:server-5425c6f7-5e8a-fe21-c000-364edceb916e Address:5425c6f7-5e8a-fe21-c000-364edceb916e}]"
2025-06-26T05:13:39.678Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:39.680Z [INFO]  server-6d2ac5a7-95b5-5825-adef-ac50da6cf1f9: entering follower state: follower="Node at 6d2ac5a7-95b5-5825-adef-ac50da6cf1f9 [Follower]" leader-address= leader-id=
2025-06-26T05:13:39.680Z [INFO]  server-32a83739-ad35-e18e-11e4-62c19c6406d5: entering follower state: follower="Node at 32a83739-ad35-e18e-11e4-62c19c6406d5 [Follower]" leader-address= leader-id=
2025-06-26T05:13:39.680Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: entering follower state: follower="Node at 5425c6f7-5e8a-fe21-c000-364edceb916e [Follower]" leader-address= leader-id=
2025-06-26T05:13:40.396Z [WARN]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:40.396Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: entering candidate state: node="Node at 5425c6f7-5e8a-fe21-c000-364edceb916e [Candidate]" term=2
2025-06-26T05:13:40.396Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: election won: term=2 tally=2
2025-06-26T05:13:40.396Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: entering leader state: leader="Node at 5425c6f7-5e8a-fe21-c000-364edceb916e [Leader]"
2025-06-26T05:13:40.396Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: added peer, starting replication: peer=server-6d2ac5a7-95b5-5825-adef-ac50da6cf1f9
2025-06-26T05:13:40.396Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: added peer, starting replication: peer=server-32a83739-ad35-e18e-11e4-62c19c6406d5
2025-06-26T05:13:40.396Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: pipelining replication: peer="{Voter server-32a83739-ad35-e18e-11e4-62c19c6406d5 32a83739-ad35-e18e-11e4-62c19c6406d5}"
2025-06-26T05:13:40.396Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: pipelining replication: peer="{Voter server-6d2ac5a7-95b5-5825-adef-ac50da6cf1f9 6d2ac5a7-95b5-5825-adef-ac50da6cf1f9}"
2025-06-26T05:13:41.402Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:40.396988083 +0000 UTC m=+48.546098401 (1 nodes), 718.36136ms from start of poll, 718.79908ms from cluster start. Timeout at 2025-06-26 05:13:41.402058763 +0000 UTC m=+49.551169081, 1.00507068s after stability
2025-06-26T05:13:41.403Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: starting snapshot up to: index=12
2025-06-26T05:13:41.403Z [INFO]  creating new snapshot: path=/tmp/raft3023955997/snapshots/2-12-1750914821403.tmp
2025-06-26T05:13:41.403Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: snapshot complete up to: index=12
2025-06-26T05:13:41.404Z [INFO]  server-64b025fe-e691-6872-5596-557f189d82fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64b025fe-e691-6872-5596-557f189d82fc Address:64b025fe-e691-6872-5596-557f189d82fc} {Suffrage:Voter ID:server-276d3250-4b29-bf73-efd5-e697f9ebf101 Address:276d3250-4b29-bf73-efd5-e697f9ebf101} {Suffrage:Voter ID:server-e48fb1a6-006f-b16e-e875-57b4620af566 Address:e48fb1a6-006f-b16e-e875-57b4620af566}]"
2025-06-26T05:13:41.404Z [INFO]  server-64b025fe-e691-6872-5596-557f189d82fc: entering follower state: follower="Node at 64b025fe-e691-6872-5596-557f189d82fc [Follower]" leader-address= leader-id=
2025-06-26T05:13:41.404Z [INFO]  server-276d3250-4b29-bf73-efd5-e697f9ebf101: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64b025fe-e691-6872-5596-557f189d82fc Address:64b025fe-e691-6872-5596-557f189d82fc} {Suffrage:Voter ID:server-276d3250-4b29-bf73-efd5-e697f9ebf101 Address:276d3250-4b29-bf73-efd5-e697f9ebf101} {Suffrage:Voter ID:server-e48fb1a6-006f-b16e-e875-57b4620af566 Address:e48fb1a6-006f-b16e-e875-57b4620af566}]"
2025-06-26T05:13:41.404Z [INFO]  server-276d3250-4b29-bf73-efd5-e697f9ebf101: entering follower state: follower="Node at 276d3250-4b29-bf73-efd5-e697f9ebf101 [Follower]" leader-address= leader-id=
2025-06-26T05:13:41.405Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64b025fe-e691-6872-5596-557f189d82fc Address:64b025fe-e691-6872-5596-557f189d82fc} {Suffrage:Voter ID:server-276d3250-4b29-bf73-efd5-e697f9ebf101 Address:276d3250-4b29-bf73-efd5-e697f9ebf101} {Suffrage:Voter ID:server-e48fb1a6-006f-b16e-e875-57b4620af566 Address:e48fb1a6-006f-b16e-e875-57b4620af566}]"
2025-06-26T05:13:41.405Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:41.405Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: entering follower state: follower="Node at e48fb1a6-006f-b16e-e875-57b4620af566 [Follower]" leader-address= leader-id=
2025-06-26T05:13:42.061Z [WARN]  server-e48fb1a6-006f-b16e-e875-57b4620af566: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:42.061Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: entering candidate state: node="Node at e48fb1a6-006f-b16e-e875-57b4620af566 [Candidate]" term=2
2025-06-26T05:13:42.061Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: election won: term=2 tally=2
2025-06-26T05:13:42.061Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: entering leader state: leader="Node at e48fb1a6-006f-b16e-e875-57b4620af566 [Leader]"
2025-06-26T05:13:42.061Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: added peer, starting replication: peer=server-64b025fe-e691-6872-5596-557f189d82fc
2025-06-26T05:13:42.061Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: added peer, starting replication: peer=server-276d3250-4b29-bf73-efd5-e697f9ebf101
2025-06-26T05:13:42.061Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: pipelining replication: peer="{Voter server-276d3250-4b29-bf73-efd5-e697f9ebf101 276d3250-4b29-bf73-efd5-e697f9ebf101}"
2025-06-26T05:13:42.061Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: pipelining replication: peer="{Voter server-64b025fe-e691-6872-5596-557f189d82fc 64b025fe-e691-6872-5596-557f189d82fc}"
2025-06-26T05:13:43.067Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:42.061703203 +0000 UTC m=+50.210813521 (1 nodes), 656.46276ms from start of poll, 657.17644ms from cluster start. Timeout at 2025-06-26 05:13:43.067518563 +0000 UTC m=+51.216628921, 1.0058154s after stability
2025-06-26T05:13:43.067Z [INFO]  creating new snapshot: path=/tmp/raft482213506/snapshots/2-15-1750914823067.tmp
2025-06-26T05:13:43.068Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: copied to local snapshot: bytes=71
2025-06-26T05:13:43.068Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: snapshot restore progress: id=2-15-1750914823067 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:43.068Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: restored user snapshot: index=15
2025-06-26T05:13:43.068Z [ERROR] server-e48fb1a6-006f-b16e-e875-57b4620af566: failed to get log: index=3 error="log not found"
2025-06-26T05:13:43.068Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: aborting pipeline replication: peer="{Voter server-276d3250-4b29-bf73-efd5-e697f9ebf101 276d3250-4b29-bf73-efd5-e697f9ebf101}"
2025-06-26T05:13:43.068Z [ERROR] server-e48fb1a6-006f-b16e-e875-57b4620af566: failed to get log: index=3 error="log not found"
2025-06-26T05:13:43.068Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: aborting pipeline replication: peer="{Voter server-64b025fe-e691-6872-5596-557f189d82fc 64b025fe-e691-6872-5596-557f189d82fc}"
2025-06-26T05:13:43.078Z [ERROR] server-e48fb1a6-006f-b16e-e875-57b4620af566: failed to get log: index=3 error="log not found"
2025-06-26T05:13:43.078Z [ERROR] server-e48fb1a6-006f-b16e-e875-57b4620af566: failed to get log: index=3 error="log not found"
2025-06-26T05:13:43.079Z [INFO]  creating new snapshot: path=/tmp/raft815121337/snapshots/2-15-1750914823079.tmp
2025-06-26T05:13:43.079Z [INFO]  creating new snapshot: path=/tmp/raft470552155/snapshots/2-15-1750914823079.tmp
2025-06-26T05:13:43.080Z [INFO]  server-64b025fe-e691-6872-5596-557f189d82fc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:43.080Z [INFO]  server-64b025fe-e691-6872-5596-557f189d82fc: copied to local snapshot: bytes=71
2025-06-26T05:13:43.080Z [INFO]  server-276d3250-4b29-bf73-efd5-e697f9ebf101: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:43.081Z [INFO]  server-64b025fe-e691-6872-5596-557f189d82fc: snapshot restore progress: id=2-15-1750914823079 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:43.081Z [INFO]  server-64b025fe-e691-6872-5596-557f189d82fc: Installed remote snapshot
2025-06-26T05:13:43.081Z [INFO]  server-276d3250-4b29-bf73-efd5-e697f9ebf101: copied to local snapshot: bytes=71
2025-06-26T05:13:43.081Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: pipelining replication: peer="{Voter server-64b025fe-e691-6872-5596-557f189d82fc 64b025fe-e691-6872-5596-557f189d82fc}"
2025-06-26T05:13:43.081Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:43.081Z [INFO]  server-276d3250-4b29-bf73-efd5-e697f9ebf101: snapshot restore progress: id=2-15-1750914823079 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:43.081Z [INFO]  server-276d3250-4b29-bf73-efd5-e697f9ebf101: Installed remote snapshot
2025-06-26T05:13:43.081Z [INFO]  server-e48fb1a6-006f-b16e-e875-57b4620af566: pipelining replication: peer="{Voter server-276d3250-4b29-bf73-efd5-e697f9ebf101 276d3250-4b29-bf73-efd5-e697f9ebf101}"
2025-06-26T05:13:44.086Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:43.081310523 +0000 UTC m=+51.230420841 (2 nodes), 1.44µs from start of poll, 1.67678376s from cluster start. Timeout at 2025-06-26 05:13:44.086521003 +0000 UTC m=+52.235631321, 1.00521048s after stability
2025-06-26T05:13:44.098Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: aborting pipeline replication: peer="{Voter server-6d2ac5a7-95b5-5825-adef-ac50da6cf1f9 6d2ac5a7-95b5-5825-adef-ac50da6cf1f9}"
2025-06-26T05:13:44.098Z [INFO]  server-5425c6f7-5e8a-fe21-c000-364edceb916e: aborting pipeline replication: peer="{Voter server-32a83739-ad35-e18e-11e4-62c19c6406d5 32a83739-ad35-e18e-11e4-62c19c6406d5}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2025-06-26T05:13:44.099Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ba57162-8bb8-ed01-2224-aebbd316ce85 Address:4ba57162-8bb8-ed01-2224-aebbd316ce85} {Suffrage:Voter ID:server-d3f37da0-3472-6fd0-d6cf-22130216ae67 Address:d3f37da0-3472-6fd0-d6cf-22130216ae67} {Suffrage:Voter ID:server-aa722938-7f66-f92e-6f16-f2bc339b72f3 Address:aa722938-7f66-f92e-6f16-f2bc339b72f3}]"
2025-06-26T05:13:44.099Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: entering follower state: follower="Node at 4ba57162-8bb8-ed01-2224-aebbd316ce85 [Follower]" leader-address= leader-id=
2025-06-26T05:13:44.099Z [INFO]  server-d3f37da0-3472-6fd0-d6cf-22130216ae67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ba57162-8bb8-ed01-2224-aebbd316ce85 Address:4ba57162-8bb8-ed01-2224-aebbd316ce85} {Suffrage:Voter ID:server-d3f37da0-3472-6fd0-d6cf-22130216ae67 Address:d3f37da0-3472-6fd0-d6cf-22130216ae67} {Suffrage:Voter ID:server-aa722938-7f66-f92e-6f16-f2bc339b72f3 Address:aa722938-7f66-f92e-6f16-f2bc339b72f3}]"
2025-06-26T05:13:44.099Z [INFO]  server-d3f37da0-3472-6fd0-d6cf-22130216ae67: entering follower state: follower="Node at d3f37da0-3472-6fd0-d6cf-22130216ae67 [Follower]" leader-address= leader-id=
2025-06-26T05:13:44.099Z [INFO]  server-aa722938-7f66-f92e-6f16-f2bc339b72f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ba57162-8bb8-ed01-2224-aebbd316ce85 Address:4ba57162-8bb8-ed01-2224-aebbd316ce85} {Suffrage:Voter ID:server-d3f37da0-3472-6fd0-d6cf-22130216ae67 Address:d3f37da0-3472-6fd0-d6cf-22130216ae67} {Suffrage:Voter ID:server-aa722938-7f66-f92e-6f16-f2bc339b72f3 Address:aa722938-7f66-f92e-6f16-f2bc339b72f3}]"
2025-06-26T05:13:44.099Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:44.099Z [INFO]  server-aa722938-7f66-f92e-6f16-f2bc339b72f3: entering follower state: follower="Node at aa722938-7f66-f92e-6f16-f2bc339b72f3 [Follower]" leader-address= leader-id=
2025-06-26T05:13:44.672Z [WARN]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:44.672Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: entering candidate state: node="Node at 4ba57162-8bb8-ed01-2224-aebbd316ce85 [Candidate]" term=2
2025-06-26T05:13:44.672Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: election won: term=2 tally=2
2025-06-26T05:13:44.673Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: entering leader state: leader="Node at 4ba57162-8bb8-ed01-2224-aebbd316ce85 [Leader]"
2025-06-26T05:13:44.673Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: added peer, starting replication: peer=server-d3f37da0-3472-6fd0-d6cf-22130216ae67
2025-06-26T05:13:44.673Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: added peer, starting replication: peer=server-aa722938-7f66-f92e-6f16-f2bc339b72f3
2025-06-26T05:13:44.673Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: pipelining replication: peer="{Voter server-aa722938-7f66-f92e-6f16-f2bc339b72f3 aa722938-7f66-f92e-6f16-f2bc339b72f3}"
2025-06-26T05:13:44.673Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: pipelining replication: peer="{Voter server-d3f37da0-3472-6fd0-d6cf-22130216ae67 d3f37da0-3472-6fd0-d6cf-22130216ae67}"
2025-06-26T05:13:45.678Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:44.673226563 +0000 UTC m=+52.822336881 (1 nodes), 573.35336ms from start of poll, 573.82048ms from cluster start. Timeout at 2025-06-26 05:13:45.678323483 +0000 UTC m=+53.827433801, 1.00509692s after stability
2025-06-26T05:13:45.679Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: starting snapshot up to: index=12
2025-06-26T05:13:45.679Z [INFO]  creating new snapshot: path=/tmp/raft998895839/snapshots/2-12-1750914825679.tmp
2025-06-26T05:13:45.679Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: snapshot complete up to: index=12
2025-06-26T05:13:45.680Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75062d76-6f01-269f-f5c8-429ccd1ef037 Address:75062d76-6f01-269f-f5c8-429ccd1ef037} {Suffrage:Voter ID:server-75ea7787-6bcf-5330-620b-9c4197d804a5 Address:75ea7787-6bcf-5330-620b-9c4197d804a5} {Suffrage:Voter ID:server-bc1f547a-759e-e313-0628-95b6356f258b Address:bc1f547a-759e-e313-0628-95b6356f258b}]"
2025-06-26T05:13:45.680Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: entering follower state: follower="Node at 75062d76-6f01-269f-f5c8-429ccd1ef037 [Follower]" leader-address= leader-id=
2025-06-26T05:13:45.680Z [INFO]  server-75ea7787-6bcf-5330-620b-9c4197d804a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75062d76-6f01-269f-f5c8-429ccd1ef037 Address:75062d76-6f01-269f-f5c8-429ccd1ef037} {Suffrage:Voter ID:server-75ea7787-6bcf-5330-620b-9c4197d804a5 Address:75ea7787-6bcf-5330-620b-9c4197d804a5} {Suffrage:Voter ID:server-bc1f547a-759e-e313-0628-95b6356f258b Address:bc1f547a-759e-e313-0628-95b6356f258b}]"
2025-06-26T05:13:45.680Z [INFO]  server-75ea7787-6bcf-5330-620b-9c4197d804a5: entering follower state: follower="Node at 75ea7787-6bcf-5330-620b-9c4197d804a5 [Follower]" leader-address= leader-id=
2025-06-26T05:13:45.680Z [INFO]  server-bc1f547a-759e-e313-0628-95b6356f258b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75062d76-6f01-269f-f5c8-429ccd1ef037 Address:75062d76-6f01-269f-f5c8-429ccd1ef037} {Suffrage:Voter ID:server-75ea7787-6bcf-5330-620b-9c4197d804a5 Address:75ea7787-6bcf-5330-620b-9c4197d804a5} {Suffrage:Voter ID:server-bc1f547a-759e-e313-0628-95b6356f258b Address:bc1f547a-759e-e313-0628-95b6356f258b}]"
2025-06-26T05:13:45.680Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:45.680Z [INFO]  server-bc1f547a-759e-e313-0628-95b6356f258b: entering follower state: follower="Node at bc1f547a-759e-e313-0628-95b6356f258b [Follower]" leader-address= leader-id=
2025-06-26T05:13:46.309Z [WARN]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:46.309Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: entering candidate state: node="Node at 75062d76-6f01-269f-f5c8-429ccd1ef037 [Candidate]" term=2
2025-06-26T05:13:46.310Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: election won: term=2 tally=2
2025-06-26T05:13:46.310Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: entering leader state: leader="Node at 75062d76-6f01-269f-f5c8-429ccd1ef037 [Leader]"
2025-06-26T05:13:46.310Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: added peer, starting replication: peer=server-75ea7787-6bcf-5330-620b-9c4197d804a5
2025-06-26T05:13:46.310Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: added peer, starting replication: peer=server-bc1f547a-759e-e313-0628-95b6356f258b
2025-06-26T05:13:46.310Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: pipelining replication: peer="{Voter server-bc1f547a-759e-e313-0628-95b6356f258b bc1f547a-759e-e313-0628-95b6356f258b}"
2025-06-26T05:13:46.310Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: pipelining replication: peer="{Voter server-75ea7787-6bcf-5330-620b-9c4197d804a5 75ea7787-6bcf-5330-620b-9c4197d804a5}"
2025-06-26T05:13:47.316Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:46.310068643 +0000 UTC m=+54.459178961 (1 nodes), 629.0898ms from start of poll, 629.5788ms from cluster start. Timeout at 2025-06-26 05:13:47.316027723 +0000 UTC m=+55.465138041, 1.00595908s after stability
2025-06-26T05:13:47.316Z [INFO]  creating new snapshot: path=/tmp/raft2815812810/snapshots/2-15-1750914827316.tmp
2025-06-26T05:13:47.316Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: copied to local snapshot: bytes=71
2025-06-26T05:13:47.317Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: snapshot restore progress: id=2-15-1750914827316 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:47.317Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: removing all old logs from log store
2025-06-26T05:13:47.317Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: compacting logs: from=1 to=2
2025-06-26T05:13:47.317Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: restored user snapshot: index=15
2025-06-26T05:13:47.317Z [ERROR] server-75062d76-6f01-269f-f5c8-429ccd1ef037: failed to get log: index=2 error="log not found"
2025-06-26T05:13:47.317Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: aborting pipeline replication: peer="{Voter server-bc1f547a-759e-e313-0628-95b6356f258b bc1f547a-759e-e313-0628-95b6356f258b}"
2025-06-26T05:13:47.317Z [ERROR] server-75062d76-6f01-269f-f5c8-429ccd1ef037: failed to get log: index=2 error="log not found"
2025-06-26T05:13:47.317Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: aborting pipeline replication: peer="{Voter server-75ea7787-6bcf-5330-620b-9c4197d804a5 75ea7787-6bcf-5330-620b-9c4197d804a5}"
2025-06-26T05:13:47.324Z [ERROR] server-75062d76-6f01-269f-f5c8-429ccd1ef037: failed to get log: index=2 error="log not found"
2025-06-26T05:13:47.324Z [INFO]  creating new snapshot: path=/tmp/raft1972548975/snapshots/2-15-1750914827324.tmp
2025-06-26T05:13:47.324Z [INFO]  server-75ea7787-6bcf-5330-620b-9c4197d804a5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:47.325Z [INFO]  server-75ea7787-6bcf-5330-620b-9c4197d804a5: copied to local snapshot: bytes=71
2025-06-26T05:13:47.325Z [INFO]  server-75ea7787-6bcf-5330-620b-9c4197d804a5: snapshot restore progress: id=2-15-1750914827324 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:47.325Z [INFO]  server-75ea7787-6bcf-5330-620b-9c4197d804a5: removing all old logs from log store
2025-06-26T05:13:47.325Z [INFO]  server-75ea7787-6bcf-5330-620b-9c4197d804a5: compacting logs: from=1 to=2
2025-06-26T05:13:47.325Z [INFO]  server-75ea7787-6bcf-5330-620b-9c4197d804a5: Installed remote snapshot
2025-06-26T05:13:47.325Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: pipelining replication: peer="{Voter server-75ea7787-6bcf-5330-620b-9c4197d804a5 75ea7787-6bcf-5330-620b-9c4197d804a5}"
2025-06-26T05:13:47.325Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:47.327Z [ERROR] server-75062d76-6f01-269f-f5c8-429ccd1ef037: failed to get log: index=2 error="log not found"
2025-06-26T05:13:47.327Z [INFO]  creating new snapshot: path=/tmp/raft2461265629/snapshots/2-15-1750914827327.tmp
2025-06-26T05:13:47.328Z [INFO]  server-bc1f547a-759e-e313-0628-95b6356f258b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:47.328Z [INFO]  server-bc1f547a-759e-e313-0628-95b6356f258b: copied to local snapshot: bytes=71
2025-06-26T05:13:47.328Z [INFO]  server-bc1f547a-759e-e313-0628-95b6356f258b: snapshot restore progress: id=2-15-1750914827327 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:47.328Z [INFO]  server-bc1f547a-759e-e313-0628-95b6356f258b: removing all old logs from log store
2025-06-26T05:13:47.328Z [INFO]  server-bc1f547a-759e-e313-0628-95b6356f258b: compacting logs: from=1 to=2
2025-06-26T05:13:47.328Z [INFO]  server-bc1f547a-759e-e313-0628-95b6356f258b: Installed remote snapshot
2025-06-26T05:13:47.328Z [INFO]  server-75062d76-6f01-269f-f5c8-429ccd1ef037: pipelining replication: peer="{Voter server-bc1f547a-759e-e313-0628-95b6356f258b bc1f547a-759e-e313-0628-95b6356f258b}"
2025-06-26T05:13:48.331Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:47.325640643 +0000 UTC m=+55.474750921 (2 nodes), 1.36µs from start of poll, 1.64515076s from cluster start. Timeout at 2025-06-26 05:13:48.330955443 +0000 UTC m=+56.480065801, 1.00531488s after stability
2025-06-26T05:13:48.342Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: aborting pipeline replication: peer="{Voter server-aa722938-7f66-f92e-6f16-f2bc339b72f3 aa722938-7f66-f92e-6f16-f2bc339b72f3}"
2025-06-26T05:13:48.342Z [INFO]  server-4ba57162-8bb8-ed01-2224-aebbd316ce85: aborting pipeline replication: peer="{Voter server-d3f37da0-3472-6fd0-d6cf-22130216ae67 d3f37da0-3472-6fd0-d6cf-22130216ae67}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2025-06-26T05:13:48.343Z [INFO]  server-6c803797-0462-2577-a7a3-d234c9391012: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c803797-0462-2577-a7a3-d234c9391012 Address:6c803797-0462-2577-a7a3-d234c9391012} {Suffrage:Voter ID:server-8965a1e4-b0b2-e46d-4806-4c694e6cc591 Address:8965a1e4-b0b2-e46d-4806-4c694e6cc591} {Suffrage:Voter ID:server-f149bc7b-19a3-adf0-4d35-f1635700eb4a Address:f149bc7b-19a3-adf0-4d35-f1635700eb4a}]"
2025-06-26T05:13:48.343Z [INFO]  server-6c803797-0462-2577-a7a3-d234c9391012: entering follower state: follower="Node at 6c803797-0462-2577-a7a3-d234c9391012 [Follower]" leader-address= leader-id=
2025-06-26T05:13:48.344Z [INFO]  server-8965a1e4-b0b2-e46d-4806-4c694e6cc591: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c803797-0462-2577-a7a3-d234c9391012 Address:6c803797-0462-2577-a7a3-d234c9391012} {Suffrage:Voter ID:server-8965a1e4-b0b2-e46d-4806-4c694e6cc591 Address:8965a1e4-b0b2-e46d-4806-4c694e6cc591} {Suffrage:Voter ID:server-f149bc7b-19a3-adf0-4d35-f1635700eb4a Address:f149bc7b-19a3-adf0-4d35-f1635700eb4a}]"
2025-06-26T05:13:48.344Z [INFO]  server-8965a1e4-b0b2-e46d-4806-4c694e6cc591: entering follower state: follower="Node at 8965a1e4-b0b2-e46d-4806-4c694e6cc591 [Follower]" leader-address= leader-id=
2025-06-26T05:13:48.344Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c803797-0462-2577-a7a3-d234c9391012 Address:6c803797-0462-2577-a7a3-d234c9391012} {Suffrage:Voter ID:server-8965a1e4-b0b2-e46d-4806-4c694e6cc591 Address:8965a1e4-b0b2-e46d-4806-4c694e6cc591} {Suffrage:Voter ID:server-f149bc7b-19a3-adf0-4d35-f1635700eb4a Address:f149bc7b-19a3-adf0-4d35-f1635700eb4a}]"
2025-06-26T05:13:48.344Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:48.344Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: entering follower state: follower="Node at f149bc7b-19a3-adf0-4d35-f1635700eb4a [Follower]" leader-address= leader-id=
2025-06-26T05:13:48.921Z [WARN]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:48.921Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: entering candidate state: node="Node at f149bc7b-19a3-adf0-4d35-f1635700eb4a [Candidate]" term=2
2025-06-26T05:13:48.921Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: election won: term=2 tally=2
2025-06-26T05:13:48.921Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: entering leader state: leader="Node at f149bc7b-19a3-adf0-4d35-f1635700eb4a [Leader]"
2025-06-26T05:13:48.921Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: added peer, starting replication: peer=server-6c803797-0462-2577-a7a3-d234c9391012
2025-06-26T05:13:48.921Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: added peer, starting replication: peer=server-8965a1e4-b0b2-e46d-4806-4c694e6cc591
2025-06-26T05:13:48.921Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: pipelining replication: peer="{Voter server-8965a1e4-b0b2-e46d-4806-4c694e6cc591 8965a1e4-b0b2-e46d-4806-4c694e6cc591}"
2025-06-26T05:13:48.922Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: pipelining replication: peer="{Voter server-6c803797-0462-2577-a7a3-d234c9391012 6c803797-0462-2577-a7a3-d234c9391012}"
2025-06-26T05:13:49.927Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:48.922364243 +0000 UTC m=+57.071474561 (1 nodes), 578.12108ms from start of poll, 578.59724ms from cluster start. Timeout at 2025-06-26 05:13:49.927446963 +0000 UTC m=+58.076557281, 1.00508272s after stability
2025-06-26T05:13:49.928Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: starting snapshot up to: index=12
2025-06-26T05:13:49.928Z [INFO]  creating new snapshot: path=/tmp/raft3228593550/snapshots/2-12-1750914829928.tmp
2025-06-26T05:13:49.928Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: snapshot complete up to: index=12
2025-06-26T05:13:49.929Z [INFO]  creating new snapshot: path=/tmp/raft3228593550/snapshots/2-113-1750914829929.tmp
2025-06-26T05:13:49.929Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: copied to local snapshot: bytes=71
2025-06-26T05:13:49.930Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: snapshot restore progress: id=2-113-1750914829929 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:49.930Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: restored user snapshot: index=113
2025-06-26T05:13:49.930Z [ERROR] server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: failed to get log: index=23 error="log not found"
2025-06-26T05:13:49.930Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: aborting pipeline replication: peer="{Voter server-8965a1e4-b0b2-e46d-4806-4c694e6cc591 8965a1e4-b0b2-e46d-4806-4c694e6cc591}"
2025-06-26T05:13:49.930Z [ERROR] server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: failed to get log: index=23 error="log not found"
2025-06-26T05:13:49.930Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: aborting pipeline replication: peer="{Voter server-6c803797-0462-2577-a7a3-d234c9391012 6c803797-0462-2577-a7a3-d234c9391012}"
2025-06-26T05:13:49.937Z [ERROR] server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: failed to get log: index=23 error="log not found"
2025-06-26T05:13:49.937Z [ERROR] server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: failed to get log: index=23 error="log not found"
2025-06-26T05:13:49.937Z [INFO]  creating new snapshot: path=/tmp/raft3156489668/snapshots/2-113-1750914829937.tmp
2025-06-26T05:13:49.938Z [INFO]  creating new snapshot: path=/tmp/raft191281664/snapshots/2-113-1750914829938.tmp
2025-06-26T05:13:49.938Z [INFO]  server-8965a1e4-b0b2-e46d-4806-4c694e6cc591: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:49.938Z [INFO]  server-6c803797-0462-2577-a7a3-d234c9391012: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:49.938Z [INFO]  server-8965a1e4-b0b2-e46d-4806-4c694e6cc591: copied to local snapshot: bytes=71
2025-06-26T05:13:49.938Z [INFO]  server-6c803797-0462-2577-a7a3-d234c9391012: copied to local snapshot: bytes=71
2025-06-26T05:13:49.938Z [INFO]  server-8965a1e4-b0b2-e46d-4806-4c694e6cc591: snapshot restore progress: id=2-113-1750914829937 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:49.938Z [INFO]  server-8965a1e4-b0b2-e46d-4806-4c694e6cc591: Installed remote snapshot
2025-06-26T05:13:49.938Z [INFO]  server-6c803797-0462-2577-a7a3-d234c9391012: snapshot restore progress: id=2-113-1750914829938 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:49.938Z [INFO]  server-6c803797-0462-2577-a7a3-d234c9391012: Installed remote snapshot
2025-06-26T05:13:49.938Z [WARN]  server-8965a1e4-b0b2-e46d-4806-4c694e6cc591: skipping application of old log: index=22
2025-06-26T05:13:49.938Z [WARN]  server-6c803797-0462-2577-a7a3-d234c9391012: skipping application of old log: index=113
2025-06-26T05:13:49.938Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: pipelining replication: peer="{Voter server-8965a1e4-b0b2-e46d-4806-4c694e6cc591 8965a1e4-b0b2-e46d-4806-4c694e6cc591}"
2025-06-26T05:13:49.938Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:49.938Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: pipelining replication: peer="{Voter server-6c803797-0462-2577-a7a3-d234c9391012 6c803797-0462-2577-a7a3-d234c9391012}"
2025-06-26T05:13:50.944Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:49.938983843 +0000 UTC m=+58.088094161 (2 nodes), 1.08µs from start of poll, 1.59521684s from cluster start. Timeout at 2025-06-26 05:13:50.944153403 +0000 UTC m=+59.093263721, 1.00516956s after stability
2025-06-26T05:13:50.956Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: aborting pipeline replication: peer="{Voter server-8965a1e4-b0b2-e46d-4806-4c694e6cc591 8965a1e4-b0b2-e46d-4806-4c694e6cc591}"
2025-06-26T05:13:50.956Z [INFO]  server-f149bc7b-19a3-adf0-4d35-f1635700eb4a: aborting pipeline replication: peer="{Voter server-6c803797-0462-2577-a7a3-d234c9391012 6c803797-0462-2577-a7a3-d234c9391012}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2025-06-26T05:13:50.957Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f49717f-5fe2-8172-625b-36f5f550a77f Address:4f49717f-5fe2-8172-625b-36f5f550a77f} {Suffrage:Voter ID:server-6e36f815-362c-25e6-1284-ab2470c1b1d7 Address:6e36f815-362c-25e6-1284-ab2470c1b1d7} {Suffrage:Voter ID:server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada Address:925e29e4-c0f4-ca84-68fc-c9cd13e07ada}]"
2025-06-26T05:13:50.957Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: entering follower state: follower="Node at 4f49717f-5fe2-8172-625b-36f5f550a77f [Follower]" leader-address= leader-id=
2025-06-26T05:13:50.957Z [INFO]  server-6e36f815-362c-25e6-1284-ab2470c1b1d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f49717f-5fe2-8172-625b-36f5f550a77f Address:4f49717f-5fe2-8172-625b-36f5f550a77f} {Suffrage:Voter ID:server-6e36f815-362c-25e6-1284-ab2470c1b1d7 Address:6e36f815-362c-25e6-1284-ab2470c1b1d7} {Suffrage:Voter ID:server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada Address:925e29e4-c0f4-ca84-68fc-c9cd13e07ada}]"
2025-06-26T05:13:50.957Z [INFO]  server-6e36f815-362c-25e6-1284-ab2470c1b1d7: entering follower state: follower="Node at 6e36f815-362c-25e6-1284-ab2470c1b1d7 [Follower]" leader-address= leader-id=
2025-06-26T05:13:50.957Z [INFO]  server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f49717f-5fe2-8172-625b-36f5f550a77f Address:4f49717f-5fe2-8172-625b-36f5f550a77f} {Suffrage:Voter ID:server-6e36f815-362c-25e6-1284-ab2470c1b1d7 Address:6e36f815-362c-25e6-1284-ab2470c1b1d7} {Suffrage:Voter ID:server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada Address:925e29e4-c0f4-ca84-68fc-c9cd13e07ada}]"
2025-06-26T05:13:50.957Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:50.957Z [INFO]  server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada: entering follower state: follower="Node at 925e29e4-c0f4-ca84-68fc-c9cd13e07ada [Follower]" leader-address= leader-id=
2025-06-26T05:13:51.505Z [WARN]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:51.505Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: entering candidate state: node="Node at 4f49717f-5fe2-8172-625b-36f5f550a77f [Candidate]" term=2
2025-06-26T05:13:51.505Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: election won: term=2 tally=2
2025-06-26T05:13:51.505Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: entering leader state: leader="Node at 4f49717f-5fe2-8172-625b-36f5f550a77f [Leader]"
2025-06-26T05:13:51.505Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: added peer, starting replication: peer=server-6e36f815-362c-25e6-1284-ab2470c1b1d7
2025-06-26T05:13:51.505Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: added peer, starting replication: peer=server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada
2025-06-26T05:13:51.505Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: pipelining replication: peer="{Voter server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada 925e29e4-c0f4-ca84-68fc-c9cd13e07ada}"
2025-06-26T05:13:51.505Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: pipelining replication: peer="{Voter server-6e36f815-362c-25e6-1284-ab2470c1b1d7 6e36f815-362c-25e6-1284-ab2470c1b1d7}"
2025-06-26T05:13:52.510Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:51.505266163 +0000 UTC m=+59.654376481 (1 nodes), 547.38992ms from start of poll, 547.91748ms from cluster start. Timeout at 2025-06-26 05:13:52.510568483 +0000 UTC m=+60.659678801, 1.00530232s after stability
2025-06-26T05:13:52.511Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: starting snapshot up to: index=12
2025-06-26T05:13:52.511Z [INFO]  creating new snapshot: path=/tmp/raft2196454514/snapshots/2-12-1750914832511.tmp
2025-06-26T05:13:52.511Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: snapshot complete up to: index=12
2025-06-26T05:13:52.512Z [INFO]  creating new snapshot: path=/tmp/raft2196454514/snapshots/2-113-1750914832512.tmp
2025-06-26T05:13:52.514Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: copied to local snapshot: bytes=71
2025-06-26T05:13:52.514Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: snapshot restore progress: id=2-113-1750914832512 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:52.515Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: removing all old logs from log store
2025-06-26T05:13:52.515Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: compacting logs: from=1 to=22
2025-06-26T05:13:52.515Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: restored user snapshot: index=113
2025-06-26T05:13:52.515Z [ERROR] server-4f49717f-5fe2-8172-625b-36f5f550a77f: failed to get log: index=22 error="log not found"
2025-06-26T05:13:52.515Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: aborting pipeline replication: peer="{Voter server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada 925e29e4-c0f4-ca84-68fc-c9cd13e07ada}"
2025-06-26T05:13:52.515Z [ERROR] server-4f49717f-5fe2-8172-625b-36f5f550a77f: failed to get log: index=22 error="log not found"
2025-06-26T05:13:52.515Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: aborting pipeline replication: peer="{Voter server-6e36f815-362c-25e6-1284-ab2470c1b1d7 6e36f815-362c-25e6-1284-ab2470c1b1d7}"
2025-06-26T05:13:52.523Z [ERROR] server-4f49717f-5fe2-8172-625b-36f5f550a77f: failed to get log: index=22 error="log not found"
2025-06-26T05:13:52.524Z [INFO]  creating new snapshot: path=/tmp/raft3826757399/snapshots/2-113-1750914832524.tmp
2025-06-26T05:13:52.524Z [INFO]  server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:52.524Z [INFO]  server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada: copied to local snapshot: bytes=71
2025-06-26T05:13:52.524Z [INFO]  server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada: snapshot restore progress: id=2-113-1750914832524 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:52.525Z [INFO]  server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada: removing all old logs from log store
2025-06-26T05:13:52.525Z [INFO]  server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada: compacting logs: from=1 to=22
2025-06-26T05:13:52.525Z [INFO]  server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada: Installed remote snapshot
2025-06-26T05:13:52.525Z [ERROR] server-4f49717f-5fe2-8172-625b-36f5f550a77f: failed to get log: index=22 error="log not found"
2025-06-26T05:13:52.525Z [WARN]  server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada: skipping application of old log: index=22
2025-06-26T05:13:52.525Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: pipelining replication: peer="{Voter server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada 925e29e4-c0f4-ca84-68fc-c9cd13e07ada}"
2025-06-26T05:13:52.525Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:52.525Z [INFO]  creating new snapshot: path=/tmp/raft886026272/snapshots/2-113-1750914832525.tmp
2025-06-26T05:13:52.525Z [INFO]  server-6e36f815-362c-25e6-1284-ab2470c1b1d7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:52.526Z [INFO]  server-6e36f815-362c-25e6-1284-ab2470c1b1d7: copied to local snapshot: bytes=71
2025-06-26T05:13:52.526Z [INFO]  server-6e36f815-362c-25e6-1284-ab2470c1b1d7: snapshot restore progress: id=2-113-1750914832525 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:52.526Z [INFO]  server-6e36f815-362c-25e6-1284-ab2470c1b1d7: removing all old logs from log store
2025-06-26T05:13:52.526Z [INFO]  server-6e36f815-362c-25e6-1284-ab2470c1b1d7: compacting logs: from=1 to=22
2025-06-26T05:13:52.526Z [INFO]  server-6e36f815-362c-25e6-1284-ab2470c1b1d7: Installed remote snapshot
2025-06-26T05:13:52.526Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: pipelining replication: peer="{Voter server-6e36f815-362c-25e6-1284-ab2470c1b1d7 6e36f815-362c-25e6-1284-ab2470c1b1d7}"
2025-06-26T05:13:53.531Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:52.525216043 +0000 UTC m=+60.674326361 (2 nodes), 1.28µs from start of poll, 1.56786736s from cluster start. Timeout at 2025-06-26 05:13:53.531034043 +0000 UTC m=+61.680144361, 1.005818s after stability
2025-06-26T05:13:53.542Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: aborting pipeline replication: peer="{Voter server-925e29e4-c0f4-ca84-68fc-c9cd13e07ada 925e29e4-c0f4-ca84-68fc-c9cd13e07ada}"
2025-06-26T05:13:53.542Z [INFO]  server-4f49717f-5fe2-8172-625b-36f5f550a77f: aborting pipeline replication: peer="{Voter server-6e36f815-362c-25e6-1284-ab2470c1b1d7 6e36f815-362c-25e6-1284-ab2470c1b1d7}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2025-06-26T05:13:53.543Z [INFO]  server-ee55f01e-5450-4b1d-9d0a-e9e301ad0278: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee55f01e-5450-4b1d-9d0a-e9e301ad0278 Address:ee55f01e-5450-4b1d-9d0a-e9e301ad0278} {Suffrage:Voter ID:server-79be5d34-4fa4-bbcf-4312-c2cac7fa9854 Address:79be5d34-4fa4-bbcf-4312-c2cac7fa9854} {Suffrage:Voter ID:server-9f17581e-d70e-9bfe-139d-ab717450cd29 Address:9f17581e-d70e-9bfe-139d-ab717450cd29}]"
2025-06-26T05:13:53.543Z [INFO]  server-79be5d34-4fa4-bbcf-4312-c2cac7fa9854: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee55f01e-5450-4b1d-9d0a-e9e301ad0278 Address:ee55f01e-5450-4b1d-9d0a-e9e301ad0278} {Suffrage:Voter ID:server-79be5d34-4fa4-bbcf-4312-c2cac7fa9854 Address:79be5d34-4fa4-bbcf-4312-c2cac7fa9854} {Suffrage:Voter ID:server-9f17581e-d70e-9bfe-139d-ab717450cd29 Address:9f17581e-d70e-9bfe-139d-ab717450cd29}]"
2025-06-26T05:13:53.544Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee55f01e-5450-4b1d-9d0a-e9e301ad0278 Address:ee55f01e-5450-4b1d-9d0a-e9e301ad0278} {Suffrage:Voter ID:server-79be5d34-4fa4-bbcf-4312-c2cac7fa9854 Address:79be5d34-4fa4-bbcf-4312-c2cac7fa9854} {Suffrage:Voter ID:server-9f17581e-d70e-9bfe-139d-ab717450cd29 Address:9f17581e-d70e-9bfe-139d-ab717450cd29}]"
2025-06-26T05:13:53.544Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:53.544Z [INFO]  server-ee55f01e-5450-4b1d-9d0a-e9e301ad0278: entering follower state: follower="Node at ee55f01e-5450-4b1d-9d0a-e9e301ad0278 [Follower]" leader-address= leader-id=
2025-06-26T05:13:53.544Z [INFO]  server-79be5d34-4fa4-bbcf-4312-c2cac7fa9854: entering follower state: follower="Node at 79be5d34-4fa4-bbcf-4312-c2cac7fa9854 [Follower]" leader-address= leader-id=
2025-06-26T05:13:53.544Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: entering follower state: follower="Node at 9f17581e-d70e-9bfe-139d-ab717450cd29 [Follower]" leader-address= leader-id=
2025-06-26T05:13:54.112Z [WARN]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:54.112Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: entering candidate state: node="Node at 9f17581e-d70e-9bfe-139d-ab717450cd29 [Candidate]" term=2
2025-06-26T05:13:54.112Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: election won: term=2 tally=2
2025-06-26T05:13:54.112Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: entering leader state: leader="Node at 9f17581e-d70e-9bfe-139d-ab717450cd29 [Leader]"
2025-06-26T05:13:54.112Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: added peer, starting replication: peer=server-ee55f01e-5450-4b1d-9d0a-e9e301ad0278
2025-06-26T05:13:54.112Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: added peer, starting replication: peer=server-79be5d34-4fa4-bbcf-4312-c2cac7fa9854
2025-06-26T05:13:54.113Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: pipelining replication: peer="{Voter server-79be5d34-4fa4-bbcf-4312-c2cac7fa9854 79be5d34-4fa4-bbcf-4312-c2cac7fa9854}"
2025-06-26T05:13:54.113Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: pipelining replication: peer="{Voter server-ee55f01e-5450-4b1d-9d0a-e9e301ad0278 ee55f01e-5450-4b1d-9d0a-e9e301ad0278}"
2025-06-26T05:13:55.118Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:54.112984043 +0000 UTC m=+62.262094361 (1 nodes), 568.87004ms from start of poll, 569.44888ms from cluster start. Timeout at 2025-06-26 05:13:55.118572323 +0000 UTC m=+63.267682681, 1.00558832s after stability
2025-06-26T05:13:55.119Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: starting snapshot up to: index=12
2025-06-26T05:13:55.119Z [INFO]  creating new snapshot: path=/tmp/raft1325588587/snapshots/2-12-1750914835119.tmp
2025-06-26T05:13:55.119Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: snapshot complete up to: index=12
2025-06-26T05:13:55.120Z [INFO]  server-9d3a0014-39e6-b719-ed35-ff8e316c9622: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d3a0014-39e6-b719-ed35-ff8e316c9622 Address:9d3a0014-39e6-b719-ed35-ff8e316c9622} {Suffrage:Voter ID:server-d83c16a2-57cd-9246-bb64-c78c7738d262 Address:d83c16a2-57cd-9246-bb64-c78c7738d262} {Suffrage:Voter ID:server-419fb716-ee27-c20a-b9df-5e92b137d8fb Address:419fb716-ee27-c20a-b9df-5e92b137d8fb}]"
2025-06-26T05:13:55.120Z [INFO]  server-9d3a0014-39e6-b719-ed35-ff8e316c9622: entering follower state: follower="Node at 9d3a0014-39e6-b719-ed35-ff8e316c9622 [Follower]" leader-address= leader-id=
2025-06-26T05:13:55.121Z [INFO]  server-d83c16a2-57cd-9246-bb64-c78c7738d262: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d3a0014-39e6-b719-ed35-ff8e316c9622 Address:9d3a0014-39e6-b719-ed35-ff8e316c9622} {Suffrage:Voter ID:server-d83c16a2-57cd-9246-bb64-c78c7738d262 Address:d83c16a2-57cd-9246-bb64-c78c7738d262} {Suffrage:Voter ID:server-419fb716-ee27-c20a-b9df-5e92b137d8fb Address:419fb716-ee27-c20a-b9df-5e92b137d8fb}]"
2025-06-26T05:13:55.121Z [INFO]  server-d83c16a2-57cd-9246-bb64-c78c7738d262: entering follower state: follower="Node at d83c16a2-57cd-9246-bb64-c78c7738d262 [Follower]" leader-address= leader-id=
2025-06-26T05:13:55.121Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d3a0014-39e6-b719-ed35-ff8e316c9622 Address:9d3a0014-39e6-b719-ed35-ff8e316c9622} {Suffrage:Voter ID:server-d83c16a2-57cd-9246-bb64-c78c7738d262 Address:d83c16a2-57cd-9246-bb64-c78c7738d262} {Suffrage:Voter ID:server-419fb716-ee27-c20a-b9df-5e92b137d8fb Address:419fb716-ee27-c20a-b9df-5e92b137d8fb}]"
2025-06-26T05:13:55.121Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:55.121Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: entering follower state: follower="Node at 419fb716-ee27-c20a-b9df-5e92b137d8fb [Follower]" leader-address= leader-id=
2025-06-26T05:13:55.681Z [WARN]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:55.681Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: entering candidate state: node="Node at 419fb716-ee27-c20a-b9df-5e92b137d8fb [Candidate]" term=2
2025-06-26T05:13:55.681Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: election won: term=2 tally=2
2025-06-26T05:13:55.681Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: entering leader state: leader="Node at 419fb716-ee27-c20a-b9df-5e92b137d8fb [Leader]"
2025-06-26T05:13:55.681Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: added peer, starting replication: peer=server-9d3a0014-39e6-b719-ed35-ff8e316c9622
2025-06-26T05:13:55.681Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: added peer, starting replication: peer=server-d83c16a2-57cd-9246-bb64-c78c7738d262
2025-06-26T05:13:55.681Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: pipelining replication: peer="{Voter server-9d3a0014-39e6-b719-ed35-ff8e316c9622 9d3a0014-39e6-b719-ed35-ff8e316c9622}"
2025-06-26T05:13:55.681Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: pipelining replication: peer="{Voter server-d83c16a2-57cd-9246-bb64-c78c7738d262 d83c16a2-57cd-9246-bb64-c78c7738d262}"
2025-06-26T05:13:56.687Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:55.681428563 +0000 UTC m=+63.830538881 (1 nodes), 560.21856ms from start of poll, 560.69736ms from cluster start. Timeout at 2025-06-26 05:13:56.686952643 +0000 UTC m=+64.836063001, 1.00552412s after stability
2025-06-26T05:13:56.687Z [INFO]  creating new snapshot: path=/tmp/raft38673720/snapshots/2-113-1750914836687.tmp
2025-06-26T05:13:56.687Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: copied to local snapshot: bytes=71
2025-06-26T05:13:56.687Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: snapshot restore progress: id=2-113-1750914836687 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:56.688Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: restored user snapshot: index=113
2025-06-26T05:13:56.688Z [ERROR] server-419fb716-ee27-c20a-b9df-5e92b137d8fb: failed to get log: index=3 error="log not found"
2025-06-26T05:13:56.688Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: aborting pipeline replication: peer="{Voter server-d83c16a2-57cd-9246-bb64-c78c7738d262 d83c16a2-57cd-9246-bb64-c78c7738d262}"
2025-06-26T05:13:56.688Z [ERROR] server-419fb716-ee27-c20a-b9df-5e92b137d8fb: failed to get log: index=3 error="log not found"
2025-06-26T05:13:56.688Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: aborting pipeline replication: peer="{Voter server-9d3a0014-39e6-b719-ed35-ff8e316c9622 9d3a0014-39e6-b719-ed35-ff8e316c9622}"
2025-06-26T05:13:56.694Z [ERROR] server-419fb716-ee27-c20a-b9df-5e92b137d8fb: failed to get log: index=3 error="log not found"
2025-06-26T05:13:56.694Z [INFO]  creating new snapshot: path=/tmp/raft961245142/snapshots/2-113-1750914836694.tmp
2025-06-26T05:13:56.694Z [INFO]  server-9d3a0014-39e6-b719-ed35-ff8e316c9622: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:56.695Z [INFO]  server-9d3a0014-39e6-b719-ed35-ff8e316c9622: copied to local snapshot: bytes=71
2025-06-26T05:13:56.695Z [INFO]  server-9d3a0014-39e6-b719-ed35-ff8e316c9622: snapshot restore progress: id=2-113-1750914836694 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:56.695Z [INFO]  server-9d3a0014-39e6-b719-ed35-ff8e316c9622: Installed remote snapshot
2025-06-26T05:13:56.695Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: pipelining replication: peer="{Voter server-9d3a0014-39e6-b719-ed35-ff8e316c9622 9d3a0014-39e6-b719-ed35-ff8e316c9622}"
2025-06-26T05:13:56.695Z [ERROR] server-419fb716-ee27-c20a-b9df-5e92b137d8fb: failed to get log: index=3 error="log not found"
2025-06-26T05:13:56.695Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:13:56.696Z [INFO]  creating new snapshot: path=/tmp/raft2113738289/snapshots/2-113-1750914836696.tmp
2025-06-26T05:13:56.696Z [INFO]  server-d83c16a2-57cd-9246-bb64-c78c7738d262: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:56.696Z [INFO]  server-d83c16a2-57cd-9246-bb64-c78c7738d262: copied to local snapshot: bytes=71
2025-06-26T05:13:56.696Z [INFO]  server-d83c16a2-57cd-9246-bb64-c78c7738d262: snapshot restore progress: id=2-113-1750914836696 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:13:56.696Z [INFO]  server-d83c16a2-57cd-9246-bb64-c78c7738d262: Installed remote snapshot
2025-06-26T05:13:56.696Z [INFO]  server-419fb716-ee27-c20a-b9df-5e92b137d8fb: pipelining replication: peer="{Voter server-d83c16a2-57cd-9246-bb64-c78c7738d262 d83c16a2-57cd-9246-bb64-c78c7738d262}"
2025-06-26T05:13:57.700Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:13:56.695771883 +0000 UTC m=+64.844882201 (2 nodes), 1.12µs from start of poll, 1.57504068s from cluster start. Timeout at 2025-06-26 05:13:57.700840763 +0000 UTC m=+65.849951081, 1.00506888s after stability
2025-06-26T05:13:57.711Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: aborting pipeline replication: peer="{Voter server-79be5d34-4fa4-bbcf-4312-c2cac7fa9854 79be5d34-4fa4-bbcf-4312-c2cac7fa9854}"
2025-06-26T05:13:57.711Z [INFO]  server-9f17581e-d70e-9bfe-139d-ab717450cd29: aborting pipeline replication: peer="{Voter server-ee55f01e-5450-4b1d-9d0a-e9e301ad0278 ee55f01e-5450-4b1d-9d0a-e9e301ad0278}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2025-06-26T05:13:57.712Z [INFO]  server-367250ff-b8b7-a93e-6a65-b2594034ae7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-367250ff-b8b7-a93e-6a65-b2594034ae7d Address:367250ff-b8b7-a93e-6a65-b2594034ae7d} {Suffrage:Voter ID:server-42ce249b-d7dc-2807-e9db-964454e56b1b Address:42ce249b-d7dc-2807-e9db-964454e56b1b} {Suffrage:Voter ID:server-eef72730-73dd-e9b4-0075-142b8c8c2df9 Address:eef72730-73dd-e9b4-0075-142b8c8c2df9}]"
2025-06-26T05:13:57.712Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-367250ff-b8b7-a93e-6a65-b2594034ae7d Address:367250ff-b8b7-a93e-6a65-b2594034ae7d} {Suffrage:Voter ID:server-42ce249b-d7dc-2807-e9db-964454e56b1b Address:42ce249b-d7dc-2807-e9db-964454e56b1b} {Suffrage:Voter ID:server-eef72730-73dd-e9b4-0075-142b8c8c2df9 Address:eef72730-73dd-e9b4-0075-142b8c8c2df9}]"
2025-06-26T05:13:57.712Z [INFO]  server-367250ff-b8b7-a93e-6a65-b2594034ae7d: entering follower state: follower="Node at 367250ff-b8b7-a93e-6a65-b2594034ae7d [Follower]" leader-address= leader-id=
2025-06-26T05:13:57.712Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: entering follower state: follower="Node at 42ce249b-d7dc-2807-e9db-964454e56b1b [Follower]" leader-address= leader-id=
2025-06-26T05:13:57.713Z [INFO]  server-eef72730-73dd-e9b4-0075-142b8c8c2df9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-367250ff-b8b7-a93e-6a65-b2594034ae7d Address:367250ff-b8b7-a93e-6a65-b2594034ae7d} {Suffrage:Voter ID:server-42ce249b-d7dc-2807-e9db-964454e56b1b Address:42ce249b-d7dc-2807-e9db-964454e56b1b} {Suffrage:Voter ID:server-eef72730-73dd-e9b4-0075-142b8c8c2df9 Address:eef72730-73dd-e9b4-0075-142b8c8c2df9}]"
2025-06-26T05:13:57.713Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:57.713Z [INFO]  server-eef72730-73dd-e9b4-0075-142b8c8c2df9: entering follower state: follower="Node at eef72730-73dd-e9b4-0075-142b8c8c2df9 [Follower]" leader-address= leader-id=
2025-06-26T05:13:58.246Z [WARN]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:58.246Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: entering candidate state: node="Node at 42ce249b-d7dc-2807-e9db-964454e56b1b [Candidate]" term=2
2025-06-26T05:13:58.246Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: election won: term=2 tally=2
2025-06-26T05:13:58.246Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: entering leader state: leader="Node at 42ce249b-d7dc-2807-e9db-964454e56b1b [Leader]"
2025-06-26T05:13:58.246Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: added peer, starting replication: peer=server-367250ff-b8b7-a93e-6a65-b2594034ae7d
2025-06-26T05:13:58.246Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: added peer, starting replication: peer=server-eef72730-73dd-e9b4-0075-142b8c8c2df9
2025-06-26T05:13:58.246Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: pipelining replication: peer="{Voter server-eef72730-73dd-e9b4-0075-142b8c8c2df9 eef72730-73dd-e9b4-0075-142b8c8c2df9}"
2025-06-26T05:13:58.246Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: pipelining replication: peer="{Voter server-367250ff-b8b7-a93e-6a65-b2594034ae7d 367250ff-b8b7-a93e-6a65-b2594034ae7d}"
2025-06-26T05:13:59.251Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:58.246785083 +0000 UTC m=+66.395895401 (1 nodes), 533.65644ms from start of poll, 534.18464ms from cluster start. Timeout at 2025-06-26 05:13:59.251848923 +0000 UTC m=+67.400959241, 1.00506384s after stability
2025-06-26T05:13:59.252Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: starting snapshot up to: index=12
2025-06-26T05:13:59.252Z [INFO]  creating new snapshot: path=/tmp/raft1078386245/snapshots/2-12-1750914839252.tmp
2025-06-26T05:13:59.253Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: snapshot complete up to: index=12
2025-06-26T05:13:59.254Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c Address:ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c} {Suffrage:Voter ID:server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8 Address:1dbaa812-c53b-6d8b-63c1-6a1ada7065f8} {Suffrage:Voter ID:server-870a16e0-ce9f-7a50-77cc-41c463da1c41 Address:870a16e0-ce9f-7a50-77cc-41c463da1c41}]"
2025-06-26T05:13:59.254Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: entering follower state: follower="Node at ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c [Follower]" leader-address= leader-id=
2025-06-26T05:13:59.254Z [INFO]  server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c Address:ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c} {Suffrage:Voter ID:server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8 Address:1dbaa812-c53b-6d8b-63c1-6a1ada7065f8} {Suffrage:Voter ID:server-870a16e0-ce9f-7a50-77cc-41c463da1c41 Address:870a16e0-ce9f-7a50-77cc-41c463da1c41}]"
2025-06-26T05:13:59.254Z [INFO]  server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8: entering follower state: follower="Node at 1dbaa812-c53b-6d8b-63c1-6a1ada7065f8 [Follower]" leader-address= leader-id=
2025-06-26T05:13:59.254Z [INFO]  server-870a16e0-ce9f-7a50-77cc-41c463da1c41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c Address:ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c} {Suffrage:Voter ID:server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8 Address:1dbaa812-c53b-6d8b-63c1-6a1ada7065f8} {Suffrage:Voter ID:server-870a16e0-ce9f-7a50-77cc-41c463da1c41 Address:870a16e0-ce9f-7a50-77cc-41c463da1c41}]"
2025-06-26T05:13:59.254Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:13:59.254Z [INFO]  server-870a16e0-ce9f-7a50-77cc-41c463da1c41: entering follower state: follower="Node at 870a16e0-ce9f-7a50-77cc-41c463da1c41 [Follower]" leader-address= leader-id=
2025-06-26T05:13:59.858Z [WARN]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:13:59.858Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: entering candidate state: node="Node at ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c [Candidate]" term=2
2025-06-26T05:13:59.858Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: election won: term=2 tally=2
2025-06-26T05:13:59.858Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: entering leader state: leader="Node at ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c [Leader]"
2025-06-26T05:13:59.858Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: added peer, starting replication: peer=server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8
2025-06-26T05:13:59.859Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: added peer, starting replication: peer=server-870a16e0-ce9f-7a50-77cc-41c463da1c41
2025-06-26T05:13:59.859Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: pipelining replication: peer="{Voter server-870a16e0-ce9f-7a50-77cc-41c463da1c41 870a16e0-ce9f-7a50-77cc-41c463da1c41}"
2025-06-26T05:13:59.859Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: pipelining replication: peer="{Voter server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8 1dbaa812-c53b-6d8b-63c1-6a1ada7065f8}"
2025-06-26T05:14:00.864Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:13:59.859138323 +0000 UTC m=+68.008248641 (1 nodes), 604.58852ms from start of poll, 605.08884ms from cluster start. Timeout at 2025-06-26 05:14:00.864545763 +0000 UTC m=+69.013656081, 1.00540744s after stability
2025-06-26T05:14:00.864Z [INFO]  creating new snapshot: path=/tmp/raft2931133894/snapshots/2-113-1750914840864.tmp
2025-06-26T05:14:00.865Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: copied to local snapshot: bytes=71
2025-06-26T05:14:00.865Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: snapshot restore progress: id=2-113-1750914840864 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:00.865Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: removing all old logs from log store
2025-06-26T05:14:00.865Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: compacting logs: from=1 to=2
2025-06-26T05:14:00.865Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: restored user snapshot: index=113
2025-06-26T05:14:00.865Z [ERROR] server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: failed to get log: index=2 error="log not found"
2025-06-26T05:14:00.865Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: aborting pipeline replication: peer="{Voter server-870a16e0-ce9f-7a50-77cc-41c463da1c41 870a16e0-ce9f-7a50-77cc-41c463da1c41}"
2025-06-26T05:14:00.865Z [ERROR] server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: failed to get log: index=2 error="log not found"
2025-06-26T05:14:00.865Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: aborting pipeline replication: peer="{Voter server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8 1dbaa812-c53b-6d8b-63c1-6a1ada7065f8}"
2025-06-26T05:14:00.874Z [ERROR] server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: failed to get log: index=2 error="log not found"
2025-06-26T05:14:00.875Z [INFO]  creating new snapshot: path=/tmp/raft2819363695/snapshots/2-113-1750914840875.tmp
2025-06-26T05:14:00.875Z [INFO]  server-870a16e0-ce9f-7a50-77cc-41c463da1c41: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:00.875Z [ERROR] server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: failed to get log: index=2 error="log not found"
2025-06-26T05:14:00.875Z [INFO]  server-870a16e0-ce9f-7a50-77cc-41c463da1c41: copied to local snapshot: bytes=71
2025-06-26T05:14:00.875Z [INFO]  creating new snapshot: path=/tmp/raft557504833/snapshots/2-113-1750914840875.tmp
2025-06-26T05:14:00.875Z [INFO]  server-870a16e0-ce9f-7a50-77cc-41c463da1c41: snapshot restore progress: id=2-113-1750914840875 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:00.876Z [INFO]  server-870a16e0-ce9f-7a50-77cc-41c463da1c41: removing all old logs from log store
2025-06-26T05:14:00.876Z [INFO]  server-870a16e0-ce9f-7a50-77cc-41c463da1c41: compacting logs: from=1 to=2
2025-06-26T05:14:00.875Z [INFO]  server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:00.876Z [INFO]  server-870a16e0-ce9f-7a50-77cc-41c463da1c41: Installed remote snapshot
2025-06-26T05:14:00.876Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: pipelining replication: peer="{Voter server-870a16e0-ce9f-7a50-77cc-41c463da1c41 870a16e0-ce9f-7a50-77cc-41c463da1c41}"
2025-06-26T05:14:00.876Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:00.876Z [INFO]  server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8: copied to local snapshot: bytes=71
2025-06-26T05:14:00.876Z [INFO]  server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8: snapshot restore progress: id=2-113-1750914840875 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:00.876Z [INFO]  server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8: removing all old logs from log store
2025-06-26T05:14:00.876Z [INFO]  server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8: compacting logs: from=1 to=2
2025-06-26T05:14:00.876Z [INFO]  server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8: Installed remote snapshot
2025-06-26T05:14:00.876Z [INFO]  server-ae2e184f-b9c0-3d8b-23ae-f0808eecbe8c: pipelining replication: peer="{Voter server-1dbaa812-c53b-6d8b-63c1-6a1ada7065f8 1dbaa812-c53b-6d8b-63c1-6a1ada7065f8}"
2025-06-26T05:14:01.882Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:00.876254683 +0000 UTC m=+69.025365001 (2 nodes), 1.2µs from start of poll, 1.6222052s from cluster start. Timeout at 2025-06-26 05:14:01.882106043 +0000 UTC m=+70.031216361, 1.00585136s after stability
2025-06-26T05:14:01.894Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: aborting pipeline replication: peer="{Voter server-eef72730-73dd-e9b4-0075-142b8c8c2df9 eef72730-73dd-e9b4-0075-142b8c8c2df9}"
2025-06-26T05:14:01.894Z [INFO]  server-42ce249b-d7dc-2807-e9db-964454e56b1b: aborting pipeline replication: peer="{Voter server-367250ff-b8b7-a93e-6a65-b2594034ae7d 367250ff-b8b7-a93e-6a65-b2594034ae7d}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2025-06-26T05:14:01.895Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df99a938-14ff-f674-e14c-b2602d379915 Address:df99a938-14ff-f674-e14c-b2602d379915} {Suffrage:Voter ID:server-0fb75c64-42f3-7c56-b448-7fd276527df5 Address:0fb75c64-42f3-7c56-b448-7fd276527df5} {Suffrage:Voter ID:server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7 Address:6c4823f1-5272-bc76-69cf-94f7f7f7c4c7}]"
2025-06-26T05:14:01.895Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: entering follower state: follower="Node at df99a938-14ff-f674-e14c-b2602d379915 [Follower]" leader-address= leader-id=
2025-06-26T05:14:01.895Z [INFO]  server-0fb75c64-42f3-7c56-b448-7fd276527df5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df99a938-14ff-f674-e14c-b2602d379915 Address:df99a938-14ff-f674-e14c-b2602d379915} {Suffrage:Voter ID:server-0fb75c64-42f3-7c56-b448-7fd276527df5 Address:0fb75c64-42f3-7c56-b448-7fd276527df5} {Suffrage:Voter ID:server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7 Address:6c4823f1-5272-bc76-69cf-94f7f7f7c4c7}]"
2025-06-26T05:14:01.896Z [INFO]  server-0fb75c64-42f3-7c56-b448-7fd276527df5: entering follower state: follower="Node at 0fb75c64-42f3-7c56-b448-7fd276527df5 [Follower]" leader-address= leader-id=
2025-06-26T05:14:01.896Z [INFO]  server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df99a938-14ff-f674-e14c-b2602d379915 Address:df99a938-14ff-f674-e14c-b2602d379915} {Suffrage:Voter ID:server-0fb75c64-42f3-7c56-b448-7fd276527df5 Address:0fb75c64-42f3-7c56-b448-7fd276527df5} {Suffrage:Voter ID:server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7 Address:6c4823f1-5272-bc76-69cf-94f7f7f7c4c7}]"
2025-06-26T05:14:01.896Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:01.896Z [INFO]  server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7: entering follower state: follower="Node at 6c4823f1-5272-bc76-69cf-94f7f7f7c4c7 [Follower]" leader-address= leader-id=
2025-06-26T05:14:02.439Z [WARN]  server-df99a938-14ff-f674-e14c-b2602d379915: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:02.439Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: entering candidate state: node="Node at df99a938-14ff-f674-e14c-b2602d379915 [Candidate]" term=2
2025-06-26T05:14:02.439Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: election won: term=2 tally=2
2025-06-26T05:14:02.439Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: entering leader state: leader="Node at df99a938-14ff-f674-e14c-b2602d379915 [Leader]"
2025-06-26T05:14:02.439Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: added peer, starting replication: peer=server-0fb75c64-42f3-7c56-b448-7fd276527df5
2025-06-26T05:14:02.439Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: added peer, starting replication: peer=server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7
2025-06-26T05:14:02.439Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: pipelining replication: peer="{Voter server-0fb75c64-42f3-7c56-b448-7fd276527df5 0fb75c64-42f3-7c56-b448-7fd276527df5}"
2025-06-26T05:14:02.439Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: pipelining replication: peer="{Voter server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7 6c4823f1-5272-bc76-69cf-94f7f7f7c4c7}"
2025-06-26T05:14:03.445Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:02.439236123 +0000 UTC m=+70.588346441 (1 nodes), 543.00044ms from start of poll, 543.63012ms from cluster start. Timeout at 2025-06-26 05:14:03.445750923 +0000 UTC m=+71.594861241, 1.0065148s after stability
2025-06-26T05:14:03.446Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: starting snapshot up to: index=12
2025-06-26T05:14:03.446Z [INFO]  creating new snapshot: path=/tmp/raft2227389376/snapshots/2-12-1750914843446.tmp
2025-06-26T05:14:03.446Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: snapshot complete up to: index=12
2025-06-26T05:14:03.447Z [INFO]  creating new snapshot: path=/tmp/raft2227389376/snapshots/2-1013-1750914843447.tmp
2025-06-26T05:14:03.448Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: copied to local snapshot: bytes=71
2025-06-26T05:14:03.448Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: snapshot restore progress: id=2-1013-1750914843447 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:03.448Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: restored user snapshot: index=1013
2025-06-26T05:14:03.448Z [ERROR] server-df99a938-14ff-f674-e14c-b2602d379915: failed to get log: index=23 error="log not found"
2025-06-26T05:14:03.448Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: aborting pipeline replication: peer="{Voter server-0fb75c64-42f3-7c56-b448-7fd276527df5 0fb75c64-42f3-7c56-b448-7fd276527df5}"
2025-06-26T05:14:03.448Z [ERROR] server-df99a938-14ff-f674-e14c-b2602d379915: failed to get log: index=23 error="log not found"
2025-06-26T05:14:03.448Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: aborting pipeline replication: peer="{Voter server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7 6c4823f1-5272-bc76-69cf-94f7f7f7c4c7}"
2025-06-26T05:14:03.448Z [ERROR] server-df99a938-14ff-f674-e14c-b2602d379915: failed to get log: index=23 error="log not found"
2025-06-26T05:14:03.448Z [INFO]  creating new snapshot: path=/tmp/raft3230653281/snapshots/2-1013-1750914843448.tmp
2025-06-26T05:14:03.451Z [INFO]  server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:03.451Z [INFO]  server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7: copied to local snapshot: bytes=71
2025-06-26T05:14:03.451Z [INFO]  server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7: snapshot restore progress: id=2-1013-1750914843448 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:03.451Z [INFO]  server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7: Installed remote snapshot
2025-06-26T05:14:03.453Z [WARN]  server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7: skipping application of old log: index=22
2025-06-26T05:14:03.453Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: pipelining replication: peer="{Voter server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7 6c4823f1-5272-bc76-69cf-94f7f7f7c4c7}"
2025-06-26T05:14:03.453Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:03.466Z [ERROR] server-df99a938-14ff-f674-e14c-b2602d379915: failed to get log: index=23 error="log not found"
2025-06-26T05:14:03.467Z [INFO]  creating new snapshot: path=/tmp/raft4159080263/snapshots/2-1013-1750914843467.tmp
2025-06-26T05:14:03.467Z [INFO]  server-0fb75c64-42f3-7c56-b448-7fd276527df5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:03.467Z [INFO]  server-0fb75c64-42f3-7c56-b448-7fd276527df5: copied to local snapshot: bytes=71
2025-06-26T05:14:03.467Z [INFO]  server-0fb75c64-42f3-7c56-b448-7fd276527df5: snapshot restore progress: id=2-1013-1750914843467 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:03.467Z [INFO]  server-0fb75c64-42f3-7c56-b448-7fd276527df5: Installed remote snapshot
2025-06-26T05:14:03.468Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: pipelining replication: peer="{Voter server-0fb75c64-42f3-7c56-b448-7fd276527df5 0fb75c64-42f3-7c56-b448-7fd276527df5}"
2025-06-26T05:14:04.461Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:03.453930483 +0000 UTC m=+71.603040761 (2 nodes), 1.32µs from start of poll, 1.55832444s from cluster start. Timeout at 2025-06-26 05:14:04.461789323 +0000 UTC m=+72.610899641, 1.00785888s after stability
2025-06-26T05:14:04.481Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: aborting pipeline replication: peer="{Voter server-6c4823f1-5272-bc76-69cf-94f7f7f7c4c7 6c4823f1-5272-bc76-69cf-94f7f7f7c4c7}"
2025-06-26T05:14:04.481Z [INFO]  server-df99a938-14ff-f674-e14c-b2602d379915: aborting pipeline replication: peer="{Voter server-0fb75c64-42f3-7c56-b448-7fd276527df5 0fb75c64-42f3-7c56-b448-7fd276527df5}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2025-06-26T05:14:04.482Z [INFO]  server-b155726b-2564-998d-11d4-4572e26fb1f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b155726b-2564-998d-11d4-4572e26fb1f4 Address:b155726b-2564-998d-11d4-4572e26fb1f4} {Suffrage:Voter ID:server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1 Address:f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1} {Suffrage:Voter ID:server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f Address:6b2a988a-0d9d-eb03-bdfb-bf94d961d31f}]"
2025-06-26T05:14:04.482Z [INFO]  server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b155726b-2564-998d-11d4-4572e26fb1f4 Address:b155726b-2564-998d-11d4-4572e26fb1f4} {Suffrage:Voter ID:server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1 Address:f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1} {Suffrage:Voter ID:server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f Address:6b2a988a-0d9d-eb03-bdfb-bf94d961d31f}]"
2025-06-26T05:14:04.482Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b155726b-2564-998d-11d4-4572e26fb1f4 Address:b155726b-2564-998d-11d4-4572e26fb1f4} {Suffrage:Voter ID:server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1 Address:f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1} {Suffrage:Voter ID:server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f Address:6b2a988a-0d9d-eb03-bdfb-bf94d961d31f}]"
2025-06-26T05:14:04.482Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:04.482Z [INFO]  server-b155726b-2564-998d-11d4-4572e26fb1f4: entering follower state: follower="Node at b155726b-2564-998d-11d4-4572e26fb1f4 [Follower]" leader-address= leader-id=
2025-06-26T05:14:04.482Z [INFO]  server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1: entering follower state: follower="Node at f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1 [Follower]" leader-address= leader-id=
2025-06-26T05:14:04.482Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: entering follower state: follower="Node at 6b2a988a-0d9d-eb03-bdfb-bf94d961d31f [Follower]" leader-address= leader-id=
2025-06-26T05:14:05.025Z [WARN]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:05.025Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: entering candidate state: node="Node at 6b2a988a-0d9d-eb03-bdfb-bf94d961d31f [Candidate]" term=2
2025-06-26T05:14:05.025Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: election won: term=2 tally=2
2025-06-26T05:14:05.025Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: entering leader state: leader="Node at 6b2a988a-0d9d-eb03-bdfb-bf94d961d31f [Leader]"
2025-06-26T05:14:05.025Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: added peer, starting replication: peer=server-b155726b-2564-998d-11d4-4572e26fb1f4
2025-06-26T05:14:05.025Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: added peer, starting replication: peer=server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1
2025-06-26T05:14:05.025Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: pipelining replication: peer="{Voter server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1 f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1}"
2025-06-26T05:14:05.026Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: pipelining replication: peer="{Voter server-b155726b-2564-998d-11d4-4572e26fb1f4 b155726b-2564-998d-11d4-4572e26fb1f4}"
2025-06-26T05:14:06.041Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:05.026155803 +0000 UTC m=+73.175266081 (1 nodes), 543.53644ms from start of poll, 543.73584ms from cluster start. Timeout at 2025-06-26 05:14:06.041603483 +0000 UTC m=+74.190713921, 1.01544784s after stability
2025-06-26T05:14:06.042Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: starting snapshot up to: index=12
2025-06-26T05:14:06.042Z [INFO]  creating new snapshot: path=/tmp/raft4126104746/snapshots/2-12-1750914846042.tmp
2025-06-26T05:14:06.042Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: snapshot complete up to: index=12
2025-06-26T05:14:06.042Z [INFO]  creating new snapshot: path=/tmp/raft4126104746/snapshots/2-1013-1750914846042.tmp
2025-06-26T05:14:06.043Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: copied to local snapshot: bytes=71
2025-06-26T05:14:06.043Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: snapshot restore progress: id=2-1013-1750914846042 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:06.043Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: removing all old logs from log store
2025-06-26T05:14:06.043Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: compacting logs: from=1 to=22
2025-06-26T05:14:06.043Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: restored user snapshot: index=1013
2025-06-26T05:14:06.043Z [ERROR] server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: failed to get log: index=22 error="log not found"
2025-06-26T05:14:06.043Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: aborting pipeline replication: peer="{Voter server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1 f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1}"
2025-06-26T05:14:06.043Z [ERROR] server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: failed to get log: index=22 error="log not found"
2025-06-26T05:14:06.043Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: aborting pipeline replication: peer="{Voter server-b155726b-2564-998d-11d4-4572e26fb1f4 b155726b-2564-998d-11d4-4572e26fb1f4}"
2025-06-26T05:14:06.053Z [ERROR] server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: failed to get log: index=22 error="log not found"
2025-06-26T05:14:06.053Z [ERROR] server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: failed to get log: index=22 error="log not found"
2025-06-26T05:14:06.053Z [INFO]  creating new snapshot: path=/tmp/raft1707052674/snapshots/2-1013-1750914846053.tmp
2025-06-26T05:14:06.053Z [INFO]  creating new snapshot: path=/tmp/raft3137104825/snapshots/2-1013-1750914846053.tmp
2025-06-26T05:14:06.054Z [INFO]  server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:06.054Z [INFO]  server-b155726b-2564-998d-11d4-4572e26fb1f4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:06.054Z [INFO]  server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1: copied to local snapshot: bytes=71
2025-06-26T05:14:06.057Z [INFO]  server-b155726b-2564-998d-11d4-4572e26fb1f4: copied to local snapshot: bytes=71
2025-06-26T05:14:06.077Z [INFO]  server-b155726b-2564-998d-11d4-4572e26fb1f4: snapshot restore progress: id=2-1013-1750914846053 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:06.077Z [INFO]  server-b155726b-2564-998d-11d4-4572e26fb1f4: removing all old logs from log store
2025-06-26T05:14:06.077Z [INFO]  server-b155726b-2564-998d-11d4-4572e26fb1f4: compacting logs: from=1 to=22
2025-06-26T05:14:06.077Z [INFO]  server-b155726b-2564-998d-11d4-4572e26fb1f4: Installed remote snapshot
2025-06-26T05:14:06.077Z [WARN]  server-b155726b-2564-998d-11d4-4572e26fb1f4: skipping application of old log: index=22
2025-06-26T05:14:06.077Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: pipelining replication: peer="{Voter server-b155726b-2564-998d-11d4-4572e26fb1f4 b155726b-2564-998d-11d4-4572e26fb1f4}"
2025-06-26T05:14:06.077Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:06.078Z [INFO]  server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1: snapshot restore progress: id=2-1013-1750914846053 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:06.078Z [INFO]  server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1: removing all old logs from log store
2025-06-26T05:14:06.078Z [INFO]  server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1: compacting logs: from=1 to=22
2025-06-26T05:14:06.078Z [INFO]  server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1: Installed remote snapshot
2025-06-26T05:14:06.078Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: pipelining replication: peer="{Voter server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1 f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1}"
2025-06-26T05:14:07.089Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:06.077914363 +0000 UTC m=+74.227024601 (2 nodes), 560ns from start of poll, 1.59549436s from cluster start. Timeout at 2025-06-26 05:14:07.089606363 +0000 UTC m=+75.238716601, 1.011692s after stability
2025-06-26T05:14:07.141Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: aborting pipeline replication: peer="{Voter server-b155726b-2564-998d-11d4-4572e26fb1f4 b155726b-2564-998d-11d4-4572e26fb1f4}"
2025-06-26T05:14:07.141Z [INFO]  server-6b2a988a-0d9d-eb03-bdfb-bf94d961d31f: aborting pipeline replication: peer="{Voter server-f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1 f06a4c7b-a458-a3f8-b20a-8734a9cdfcf1}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2025-06-26T05:14:07.142Z [INFO]  server-b708ebd3-de4e-0633-cf2d-664900593e0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b708ebd3-de4e-0633-cf2d-664900593e0d Address:b708ebd3-de4e-0633-cf2d-664900593e0d} {Suffrage:Voter ID:server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d Address:676f9326-fe12-e9f3-2a84-b1bf7a3b851d} {Suffrage:Voter ID:server-0b012fb2-95b5-8fdc-70b6-06584a9559b8 Address:0b012fb2-95b5-8fdc-70b6-06584a9559b8}]"
2025-06-26T05:14:07.142Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b708ebd3-de4e-0633-cf2d-664900593e0d Address:b708ebd3-de4e-0633-cf2d-664900593e0d} {Suffrage:Voter ID:server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d Address:676f9326-fe12-e9f3-2a84-b1bf7a3b851d} {Suffrage:Voter ID:server-0b012fb2-95b5-8fdc-70b6-06584a9559b8 Address:0b012fb2-95b5-8fdc-70b6-06584a9559b8}]"
2025-06-26T05:14:07.142Z [INFO]  server-0b012fb2-95b5-8fdc-70b6-06584a9559b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b708ebd3-de4e-0633-cf2d-664900593e0d Address:b708ebd3-de4e-0633-cf2d-664900593e0d} {Suffrage:Voter ID:server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d Address:676f9326-fe12-e9f3-2a84-b1bf7a3b851d} {Suffrage:Voter ID:server-0b012fb2-95b5-8fdc-70b6-06584a9559b8 Address:0b012fb2-95b5-8fdc-70b6-06584a9559b8}]"
2025-06-26T05:14:07.142Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:07.142Z [INFO]  server-b708ebd3-de4e-0633-cf2d-664900593e0d: entering follower state: follower="Node at b708ebd3-de4e-0633-cf2d-664900593e0d [Follower]" leader-address= leader-id=
2025-06-26T05:14:07.142Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: entering follower state: follower="Node at 676f9326-fe12-e9f3-2a84-b1bf7a3b851d [Follower]" leader-address= leader-id=
2025-06-26T05:14:07.142Z [INFO]  server-0b012fb2-95b5-8fdc-70b6-06584a9559b8: entering follower state: follower="Node at 0b012fb2-95b5-8fdc-70b6-06584a9559b8 [Follower]" leader-address= leader-id=
2025-06-26T05:14:07.917Z [WARN]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:07.917Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: entering candidate state: node="Node at 676f9326-fe12-e9f3-2a84-b1bf7a3b851d [Candidate]" term=2
2025-06-26T05:14:07.917Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: election won: term=2 tally=2
2025-06-26T05:14:07.917Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: entering leader state: leader="Node at 676f9326-fe12-e9f3-2a84-b1bf7a3b851d [Leader]"
2025-06-26T05:14:07.917Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: added peer, starting replication: peer=server-b708ebd3-de4e-0633-cf2d-664900593e0d
2025-06-26T05:14:07.917Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: added peer, starting replication: peer=server-0b012fb2-95b5-8fdc-70b6-06584a9559b8
2025-06-26T05:14:07.917Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: pipelining replication: peer="{Voter server-0b012fb2-95b5-8fdc-70b6-06584a9559b8 0b012fb2-95b5-8fdc-70b6-06584a9559b8}"
2025-06-26T05:14:07.918Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: pipelining replication: peer="{Voter server-b708ebd3-de4e-0633-cf2d-664900593e0d b708ebd3-de4e-0633-cf2d-664900593e0d}"
2025-06-26T05:14:08.938Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:07.918070483 +0000 UTC m=+76.067180761 (1 nodes), 775.46368ms from start of poll, 775.66616ms from cluster start. Timeout at 2025-06-26 05:14:08.925684083 +0000 UTC m=+77.074794361, 1.0076136s after stability
2025-06-26T05:14:08.938Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: starting snapshot up to: index=12
2025-06-26T05:14:08.938Z [INFO]  creating new snapshot: path=/tmp/raft743815967/snapshots/2-12-1750914848938.tmp
2025-06-26T05:14:08.938Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: snapshot complete up to: index=12
2025-06-26T05:14:08.939Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b09e9f8-4fcf-e553-8c25-770083b273e6 Address:5b09e9f8-4fcf-e553-8c25-770083b273e6} {Suffrage:Voter ID:server-a1e22054-734e-08ca-fba0-a19a6098f49c Address:a1e22054-734e-08ca-fba0-a19a6098f49c} {Suffrage:Voter ID:server-0f122e01-a0a1-ad42-8130-ce8355a5f67f Address:0f122e01-a0a1-ad42-8130-ce8355a5f67f}]"
2025-06-26T05:14:08.939Z [INFO]  server-a1e22054-734e-08ca-fba0-a19a6098f49c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b09e9f8-4fcf-e553-8c25-770083b273e6 Address:5b09e9f8-4fcf-e553-8c25-770083b273e6} {Suffrage:Voter ID:server-a1e22054-734e-08ca-fba0-a19a6098f49c Address:a1e22054-734e-08ca-fba0-a19a6098f49c} {Suffrage:Voter ID:server-0f122e01-a0a1-ad42-8130-ce8355a5f67f Address:0f122e01-a0a1-ad42-8130-ce8355a5f67f}]"
2025-06-26T05:14:08.939Z [INFO]  server-0f122e01-a0a1-ad42-8130-ce8355a5f67f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b09e9f8-4fcf-e553-8c25-770083b273e6 Address:5b09e9f8-4fcf-e553-8c25-770083b273e6} {Suffrage:Voter ID:server-a1e22054-734e-08ca-fba0-a19a6098f49c Address:a1e22054-734e-08ca-fba0-a19a6098f49c} {Suffrage:Voter ID:server-0f122e01-a0a1-ad42-8130-ce8355a5f67f Address:0f122e01-a0a1-ad42-8130-ce8355a5f67f}]"
2025-06-26T05:14:08.939Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:08.939Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: entering follower state: follower="Node at 5b09e9f8-4fcf-e553-8c25-770083b273e6 [Follower]" leader-address= leader-id=
2025-06-26T05:14:08.939Z [INFO]  server-a1e22054-734e-08ca-fba0-a19a6098f49c: entering follower state: follower="Node at a1e22054-734e-08ca-fba0-a19a6098f49c [Follower]" leader-address= leader-id=
2025-06-26T05:14:08.939Z [INFO]  server-0f122e01-a0a1-ad42-8130-ce8355a5f67f: entering follower state: follower="Node at 0f122e01-a0a1-ad42-8130-ce8355a5f67f [Follower]" leader-address= leader-id=
2025-06-26T05:14:09.485Z [WARN]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:09.485Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: entering candidate state: node="Node at 5b09e9f8-4fcf-e553-8c25-770083b273e6 [Candidate]" term=2
2025-06-26T05:14:09.485Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: election won: term=2 tally=2
2025-06-26T05:14:09.485Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: entering leader state: leader="Node at 5b09e9f8-4fcf-e553-8c25-770083b273e6 [Leader]"
2025-06-26T05:14:09.485Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: added peer, starting replication: peer=server-a1e22054-734e-08ca-fba0-a19a6098f49c
2025-06-26T05:14:09.485Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: added peer, starting replication: peer=server-0f122e01-a0a1-ad42-8130-ce8355a5f67f
2025-06-26T05:14:09.485Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: pipelining replication: peer="{Voter server-0f122e01-a0a1-ad42-8130-ce8355a5f67f 0f122e01-a0a1-ad42-8130-ce8355a5f67f}"
2025-06-26T05:14:09.486Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: pipelining replication: peer="{Voter server-a1e22054-734e-08ca-fba0-a19a6098f49c a1e22054-734e-08ca-fba0-a19a6098f49c}"
2025-06-26T05:14:10.493Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:09.485999363 +0000 UTC m=+77.635109641 (1 nodes), 546.4946ms from start of poll, 546.66332ms from cluster start. Timeout at 2025-06-26 05:14:10.493790243 +0000 UTC m=+78.642900481, 1.00779084s after stability
2025-06-26T05:14:10.494Z [INFO]  creating new snapshot: path=/tmp/raft1239560160/snapshots/2-1013-1750914850494.tmp
2025-06-26T05:14:10.494Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: copied to local snapshot: bytes=71
2025-06-26T05:14:10.494Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: snapshot restore progress: id=2-1013-1750914850494 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:10.494Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: restored user snapshot: index=1013
2025-06-26T05:14:10.494Z [ERROR] server-5b09e9f8-4fcf-e553-8c25-770083b273e6: failed to get log: index=3 error="log not found"
2025-06-26T05:14:10.494Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: aborting pipeline replication: peer="{Voter server-0f122e01-a0a1-ad42-8130-ce8355a5f67f 0f122e01-a0a1-ad42-8130-ce8355a5f67f}"
2025-06-26T05:14:10.494Z [ERROR] server-5b09e9f8-4fcf-e553-8c25-770083b273e6: failed to get log: index=3 error="log not found"
2025-06-26T05:14:10.494Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: aborting pipeline replication: peer="{Voter server-a1e22054-734e-08ca-fba0-a19a6098f49c a1e22054-734e-08ca-fba0-a19a6098f49c}"
2025-06-26T05:14:10.502Z [ERROR] server-5b09e9f8-4fcf-e553-8c25-770083b273e6: failed to get log: index=3 error="log not found"
2025-06-26T05:14:10.502Z [INFO]  creating new snapshot: path=/tmp/raft1520369274/snapshots/2-1013-1750914850502.tmp
2025-06-26T05:14:10.502Z [INFO]  server-a1e22054-734e-08ca-fba0-a19a6098f49c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:10.502Z [INFO]  server-a1e22054-734e-08ca-fba0-a19a6098f49c: copied to local snapshot: bytes=71
2025-06-26T05:14:10.502Z [INFO]  server-a1e22054-734e-08ca-fba0-a19a6098f49c: snapshot restore progress: id=2-1013-1750914850502 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:10.502Z [INFO]  server-a1e22054-734e-08ca-fba0-a19a6098f49c: Installed remote snapshot
2025-06-26T05:14:10.502Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: pipelining replication: peer="{Voter server-a1e22054-734e-08ca-fba0-a19a6098f49c a1e22054-734e-08ca-fba0-a19a6098f49c}"
2025-06-26T05:14:10.502Z [ERROR] server-5b09e9f8-4fcf-e553-8c25-770083b273e6: failed to get log: index=3 error="log not found"
2025-06-26T05:14:10.502Z [INFO]  creating new snapshot: path=/tmp/raft3279552522/snapshots/2-1013-1750914850502.tmp
2025-06-26T05:14:10.503Z [INFO]  server-0f122e01-a0a1-ad42-8130-ce8355a5f67f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:10.503Z [INFO]  server-0f122e01-a0a1-ad42-8130-ce8355a5f67f: copied to local snapshot: bytes=71
2025-06-26T05:14:10.503Z [INFO]  server-0f122e01-a0a1-ad42-8130-ce8355a5f67f: snapshot restore progress: id=2-1013-1750914850502 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:10.503Z [INFO]  server-0f122e01-a0a1-ad42-8130-ce8355a5f67f: Installed remote snapshot
2025-06-26T05:14:10.503Z [INFO]  server-5b09e9f8-4fcf-e553-8c25-770083b273e6: pipelining replication: peer="{Voter server-0f122e01-a0a1-ad42-8130-ce8355a5f67f 0f122e01-a0a1-ad42-8130-ce8355a5f67f}"
2025-06-26T05:14:10.503Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:11.510Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:10.503479403 +0000 UTC m=+78.652589641 (2 nodes), 400ns from start of poll, 1.56414332s from cluster start. Timeout at 2025-06-26 05:14:11.510461443 +0000 UTC m=+79.659571721, 1.00698208s after stability
2025-06-26T05:14:11.525Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: aborting pipeline replication: peer="{Voter server-0b012fb2-95b5-8fdc-70b6-06584a9559b8 0b012fb2-95b5-8fdc-70b6-06584a9559b8}"
2025-06-26T05:14:11.525Z [INFO]  server-676f9326-fe12-e9f3-2a84-b1bf7a3b851d: aborting pipeline replication: peer="{Voter server-b708ebd3-de4e-0633-cf2d-664900593e0d b708ebd3-de4e-0633-cf2d-664900593e0d}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2025-06-26T05:14:11.526Z [INFO]  server-0671c998-1060-6229-dea0-0f5a7f084111: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0671c998-1060-6229-dea0-0f5a7f084111 Address:0671c998-1060-6229-dea0-0f5a7f084111} {Suffrage:Voter ID:server-a7c7cddc-0d74-877e-c02a-98793193ef71 Address:a7c7cddc-0d74-877e-c02a-98793193ef71} {Suffrage:Voter ID:server-7fea760b-de44-d1cc-de50-b55edf5bea64 Address:7fea760b-de44-d1cc-de50-b55edf5bea64}]"
2025-06-26T05:14:11.526Z [INFO]  server-a7c7cddc-0d74-877e-c02a-98793193ef71: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0671c998-1060-6229-dea0-0f5a7f084111 Address:0671c998-1060-6229-dea0-0f5a7f084111} {Suffrage:Voter ID:server-a7c7cddc-0d74-877e-c02a-98793193ef71 Address:a7c7cddc-0d74-877e-c02a-98793193ef71} {Suffrage:Voter ID:server-7fea760b-de44-d1cc-de50-b55edf5bea64 Address:7fea760b-de44-d1cc-de50-b55edf5bea64}]"
2025-06-26T05:14:11.526Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0671c998-1060-6229-dea0-0f5a7f084111 Address:0671c998-1060-6229-dea0-0f5a7f084111} {Suffrage:Voter ID:server-a7c7cddc-0d74-877e-c02a-98793193ef71 Address:a7c7cddc-0d74-877e-c02a-98793193ef71} {Suffrage:Voter ID:server-7fea760b-de44-d1cc-de50-b55edf5bea64 Address:7fea760b-de44-d1cc-de50-b55edf5bea64}]"
2025-06-26T05:14:11.526Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:11.526Z [INFO]  server-0671c998-1060-6229-dea0-0f5a7f084111: entering follower state: follower="Node at 0671c998-1060-6229-dea0-0f5a7f084111 [Follower]" leader-address= leader-id=
2025-06-26T05:14:11.526Z [INFO]  server-a7c7cddc-0d74-877e-c02a-98793193ef71: entering follower state: follower="Node at a7c7cddc-0d74-877e-c02a-98793193ef71 [Follower]" leader-address= leader-id=
2025-06-26T05:14:11.526Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: entering follower state: follower="Node at 7fea760b-de44-d1cc-de50-b55edf5bea64 [Follower]" leader-address= leader-id=
2025-06-26T05:14:12.048Z [WARN]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:12.048Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: entering candidate state: node="Node at 7fea760b-de44-d1cc-de50-b55edf5bea64 [Candidate]" term=2
2025-06-26T05:14:12.048Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: election won: term=2 tally=2
2025-06-26T05:14:12.048Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: entering leader state: leader="Node at 7fea760b-de44-d1cc-de50-b55edf5bea64 [Leader]"
2025-06-26T05:14:12.048Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: added peer, starting replication: peer=server-0671c998-1060-6229-dea0-0f5a7f084111
2025-06-26T05:14:12.048Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: added peer, starting replication: peer=server-a7c7cddc-0d74-877e-c02a-98793193ef71
2025-06-26T05:14:12.048Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: pipelining replication: peer="{Voter server-a7c7cddc-0d74-877e-c02a-98793193ef71 a7c7cddc-0d74-877e-c02a-98793193ef71}"
2025-06-26T05:14:12.048Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: pipelining replication: peer="{Voter server-0671c998-1060-6229-dea0-0f5a7f084111 0671c998-1060-6229-dea0-0f5a7f084111}"
2025-06-26T05:14:13.053Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:12.048756803 +0000 UTC m=+80.197867041 (1 nodes), 522.33336ms from start of poll, 522.5278ms from cluster start. Timeout at 2025-06-26 05:14:13.053962963 +0000 UTC m=+81.203073241, 1.0052062s after stability
2025-06-26T05:14:13.054Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: starting snapshot up to: index=12
2025-06-26T05:14:13.054Z [INFO]  creating new snapshot: path=/tmp/raft3328176820/snapshots/2-12-1750914853054.tmp
2025-06-26T05:14:13.054Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: snapshot complete up to: index=12
2025-06-26T05:14:13.055Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9d75231-5a14-17b2-2737-c48c632d9ae5 Address:f9d75231-5a14-17b2-2737-c48c632d9ae5} {Suffrage:Voter ID:server-45695aad-7d7a-a067-bdb4-f2fd34bfa828 Address:45695aad-7d7a-a067-bdb4-f2fd34bfa828} {Suffrage:Voter ID:server-6640465b-254e-2934-ee44-768d7e2cb123 Address:6640465b-254e-2934-ee44-768d7e2cb123}]"
2025-06-26T05:14:13.055Z [INFO]  server-45695aad-7d7a-a067-bdb4-f2fd34bfa828: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9d75231-5a14-17b2-2737-c48c632d9ae5 Address:f9d75231-5a14-17b2-2737-c48c632d9ae5} {Suffrage:Voter ID:server-45695aad-7d7a-a067-bdb4-f2fd34bfa828 Address:45695aad-7d7a-a067-bdb4-f2fd34bfa828} {Suffrage:Voter ID:server-6640465b-254e-2934-ee44-768d7e2cb123 Address:6640465b-254e-2934-ee44-768d7e2cb123}]"
2025-06-26T05:14:13.055Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9d75231-5a14-17b2-2737-c48c632d9ae5 Address:f9d75231-5a14-17b2-2737-c48c632d9ae5} {Suffrage:Voter ID:server-45695aad-7d7a-a067-bdb4-f2fd34bfa828 Address:45695aad-7d7a-a067-bdb4-f2fd34bfa828} {Suffrage:Voter ID:server-6640465b-254e-2934-ee44-768d7e2cb123 Address:6640465b-254e-2934-ee44-768d7e2cb123}]"
2025-06-26T05:14:13.055Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:13.055Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: entering follower state: follower="Node at f9d75231-5a14-17b2-2737-c48c632d9ae5 [Follower]" leader-address= leader-id=
2025-06-26T05:14:13.055Z [INFO]  server-45695aad-7d7a-a067-bdb4-f2fd34bfa828: entering follower state: follower="Node at 45695aad-7d7a-a067-bdb4-f2fd34bfa828 [Follower]" leader-address= leader-id=
2025-06-26T05:14:13.055Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: entering follower state: follower="Node at 6640465b-254e-2934-ee44-768d7e2cb123 [Follower]" leader-address= leader-id=
2025-06-26T05:14:13.762Z [WARN]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:13.762Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: entering candidate state: node="Node at f9d75231-5a14-17b2-2737-c48c632d9ae5 [Candidate]" term=2
2025-06-26T05:14:13.762Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: election won: term=2 tally=2
2025-06-26T05:14:13.762Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: entering leader state: leader="Node at f9d75231-5a14-17b2-2737-c48c632d9ae5 [Leader]"
2025-06-26T05:14:13.762Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: added peer, starting replication: peer=server-45695aad-7d7a-a067-bdb4-f2fd34bfa828
2025-06-26T05:14:13.762Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: added peer, starting replication: peer=server-6640465b-254e-2934-ee44-768d7e2cb123
2025-06-26T05:14:13.762Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: pipelining replication: peer="{Voter server-45695aad-7d7a-a067-bdb4-f2fd34bfa828 45695aad-7d7a-a067-bdb4-f2fd34bfa828}"
2025-06-26T05:14:13.765Z [WARN]  server-6640465b-254e-2934-ee44-768d7e2cb123: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:13.765Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: entering candidate state: node="Node at 6640465b-254e-2934-ee44-768d7e2cb123 [Candidate]" term=2
2025-06-26T05:14:13.765Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: duplicate requestVote for same term: term=2
2025-06-26T05:14:13.765Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: entering follower state: follower="Node at 6640465b-254e-2934-ee44-768d7e2cb123 [Follower]" leader-address=f9d75231-5a14-17b2-2737-c48c632d9ae5 leader-id=server-f9d75231-5a14-17b2-2737-c48c632d9ae5
2025-06-26T05:14:13.765Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: pipelining replication: peer="{Voter server-6640465b-254e-2934-ee44-768d7e2cb123 6640465b-254e-2934-ee44-768d7e2cb123}"
2025-06-26T05:14:13.766Z [WARN]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: rejecting vote request since we have a leader: from=6640465b-254e-2934-ee44-768d7e2cb123 leader=f9d75231-5a14-17b2-2737-c48c632d9ae5 leader-id=server-f9d75231-5a14-17b2-2737-c48c632d9ae5
2025-06-26T05:14:13.766Z [WARN]  server-45695aad-7d7a-a067-bdb4-f2fd34bfa828: rejecting vote request since we have a leader: from=6640465b-254e-2934-ee44-768d7e2cb123 leader=f9d75231-5a14-17b2-2737-c48c632d9ae5 leader-id=server-f9d75231-5a14-17b2-2737-c48c632d9ae5
2025-06-26T05:14:14.777Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:13.766154523 +0000 UTC m=+81.915264761 (1 nodes), 710.66936ms from start of poll, 710.84816ms from cluster start. Timeout at 2025-06-26 05:14:14.777600563 +0000 UTC m=+82.926710801, 1.01144604s after stability
2025-06-26T05:14:14.777Z [INFO]  creating new snapshot: path=/tmp/raft1001696183/snapshots/2-1013-1750914854777.tmp
2025-06-26T05:14:14.778Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: copied to local snapshot: bytes=71
2025-06-26T05:14:14.778Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: snapshot restore progress: id=2-1013-1750914854777 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:14.778Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: removing all old logs from log store
2025-06-26T05:14:14.778Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: compacting logs: from=1 to=2
2025-06-26T05:14:14.778Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: restored user snapshot: index=1013
2025-06-26T05:14:14.778Z [ERROR] server-f9d75231-5a14-17b2-2737-c48c632d9ae5: failed to get log: index=2 error="log not found"
2025-06-26T05:14:14.778Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: aborting pipeline replication: peer="{Voter server-6640465b-254e-2934-ee44-768d7e2cb123 6640465b-254e-2934-ee44-768d7e2cb123}"
2025-06-26T05:14:14.778Z [ERROR] server-f9d75231-5a14-17b2-2737-c48c632d9ae5: failed to get log: index=2 error="log not found"
2025-06-26T05:14:14.778Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: aborting pipeline replication: peer="{Voter server-45695aad-7d7a-a067-bdb4-f2fd34bfa828 45695aad-7d7a-a067-bdb4-f2fd34bfa828}"
2025-06-26T05:14:14.789Z [ERROR] server-f9d75231-5a14-17b2-2737-c48c632d9ae5: failed to get log: index=2 error="log not found"
2025-06-26T05:14:14.789Z [ERROR] server-f9d75231-5a14-17b2-2737-c48c632d9ae5: failed to get log: index=2 error="log not found"
2025-06-26T05:14:14.789Z [INFO]  creating new snapshot: path=/tmp/raft2237793497/snapshots/2-1013-1750914854789.tmp
2025-06-26T05:14:14.789Z [INFO]  server-45695aad-7d7a-a067-bdb4-f2fd34bfa828: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:14.790Z [INFO]  server-45695aad-7d7a-a067-bdb4-f2fd34bfa828: copied to local snapshot: bytes=71
2025-06-26T05:14:14.790Z [INFO]  server-45695aad-7d7a-a067-bdb4-f2fd34bfa828: snapshot restore progress: id=2-1013-1750914854789 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:14.790Z [INFO]  server-45695aad-7d7a-a067-bdb4-f2fd34bfa828: removing all old logs from log store
2025-06-26T05:14:14.790Z [INFO]  server-45695aad-7d7a-a067-bdb4-f2fd34bfa828: compacting logs: from=1 to=2
2025-06-26T05:14:14.790Z [INFO]  server-45695aad-7d7a-a067-bdb4-f2fd34bfa828: Installed remote snapshot
2025-06-26T05:14:14.790Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: pipelining replication: peer="{Voter server-45695aad-7d7a-a067-bdb4-f2fd34bfa828 45695aad-7d7a-a067-bdb4-f2fd34bfa828}"
2025-06-26T05:14:14.790Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:14.793Z [INFO]  creating new snapshot: path=/tmp/raft4133320413/snapshots/2-1013-1750914854793.tmp
2025-06-26T05:14:14.793Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:14.794Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: copied to local snapshot: bytes=71
2025-06-26T05:14:14.794Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: snapshot restore progress: id=2-1013-1750914854793 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:14.794Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: removing all old logs from log store
2025-06-26T05:14:14.794Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: compacting logs: from=1 to=2
2025-06-26T05:14:14.794Z [INFO]  server-6640465b-254e-2934-ee44-768d7e2cb123: Installed remote snapshot
2025-06-26T05:14:14.794Z [INFO]  server-f9d75231-5a14-17b2-2737-c48c632d9ae5: pipelining replication: peer="{Voter server-6640465b-254e-2934-ee44-768d7e2cb123 6640465b-254e-2934-ee44-768d7e2cb123}"
2025-06-26T05:14:15.801Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:14.790352963 +0000 UTC m=+82.939463201 (2 nodes), 440ns from start of poll, 1.7350466s from cluster start. Timeout at 2025-06-26 05:14:15.801599523 +0000 UTC m=+83.950709801, 1.0112466s after stability
2025-06-26T05:14:15.821Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: aborting pipeline replication: peer="{Voter server-a7c7cddc-0d74-877e-c02a-98793193ef71 a7c7cddc-0d74-877e-c02a-98793193ef71}"
2025-06-26T05:14:15.822Z [INFO]  server-7fea760b-de44-d1cc-de50-b55edf5bea64: aborting pipeline replication: peer="{Voter server-0671c998-1060-6229-dea0-0f5a7f084111 0671c998-1060-6229-dea0-0f5a7f084111}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2025-06-26T05:14:15.822Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f Address:b1b7da20-e303-20cd-2faa-cc2fe3c2823f} {Suffrage:Voter ID:server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc Address:e5eccde2-6b1b-0030-bbee-a8e7626df4bc} {Suffrage:Voter ID:server-f5786679-9a7b-64d5-1a72-092063626a84 Address:f5786679-9a7b-64d5-1a72-092063626a84}]"
2025-06-26T05:14:15.822Z [INFO]  server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f Address:b1b7da20-e303-20cd-2faa-cc2fe3c2823f} {Suffrage:Voter ID:server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc Address:e5eccde2-6b1b-0030-bbee-a8e7626df4bc} {Suffrage:Voter ID:server-f5786679-9a7b-64d5-1a72-092063626a84 Address:f5786679-9a7b-64d5-1a72-092063626a84}]"
2025-06-26T05:14:15.822Z [INFO]  server-f5786679-9a7b-64d5-1a72-092063626a84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f Address:b1b7da20-e303-20cd-2faa-cc2fe3c2823f} {Suffrage:Voter ID:server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc Address:e5eccde2-6b1b-0030-bbee-a8e7626df4bc} {Suffrage:Voter ID:server-f5786679-9a7b-64d5-1a72-092063626a84 Address:f5786679-9a7b-64d5-1a72-092063626a84}]"
2025-06-26T05:14:15.822Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:15.822Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: entering follower state: follower="Node at b1b7da20-e303-20cd-2faa-cc2fe3c2823f [Follower]" leader-address= leader-id=
2025-06-26T05:14:15.822Z [INFO]  server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc: entering follower state: follower="Node at e5eccde2-6b1b-0030-bbee-a8e7626df4bc [Follower]" leader-address= leader-id=
2025-06-26T05:14:15.822Z [INFO]  server-f5786679-9a7b-64d5-1a72-092063626a84: entering follower state: follower="Node at f5786679-9a7b-64d5-1a72-092063626a84 [Follower]" leader-address= leader-id=
2025-06-26T05:14:16.389Z [WARN]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:16.389Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: entering candidate state: node="Node at b1b7da20-e303-20cd-2faa-cc2fe3c2823f [Candidate]" term=2
2025-06-26T05:14:16.389Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: election won: term=2 tally=2
2025-06-26T05:14:16.389Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: entering leader state: leader="Node at b1b7da20-e303-20cd-2faa-cc2fe3c2823f [Leader]"
2025-06-26T05:14:16.389Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: added peer, starting replication: peer=server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc
2025-06-26T05:14:16.389Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: added peer, starting replication: peer=server-f5786679-9a7b-64d5-1a72-092063626a84
2025-06-26T05:14:16.389Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: pipelining replication: peer="{Voter server-f5786679-9a7b-64d5-1a72-092063626a84 f5786679-9a7b-64d5-1a72-092063626a84}"
2025-06-26T05:14:16.389Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: pipelining replication: peer="{Voter server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc e5eccde2-6b1b-0030-bbee-a8e7626df4bc}"
2025-06-26T05:14:17.401Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:16.389787483 +0000 UTC m=+84.538897761 (1 nodes), 567.06472ms from start of poll, 567.29432ms from cluster start. Timeout at 2025-06-26 05:14:17.401582483 +0000 UTC m=+85.550692761, 1.011795s after stability
2025-06-26T05:14:17.401Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: starting snapshot up to: index=12
2025-06-26T05:14:17.402Z [INFO]  creating new snapshot: path=/tmp/raft5943598/snapshots/2-12-1750914857402.tmp
2025-06-26T05:14:17.402Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: snapshot complete up to: index=12
2025-06-26T05:14:17.402Z [INFO]  creating new snapshot: path=/tmp/raft5943598/snapshots/2-10013-1750914857402.tmp
2025-06-26T05:14:17.403Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: copied to local snapshot: bytes=71
2025-06-26T05:14:17.403Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: snapshot restore progress: id=2-10013-1750914857402 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:17.403Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: restored user snapshot: index=10013
2025-06-26T05:14:17.403Z [ERROR] server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: failed to get log: index=23 error="log not found"
2025-06-26T05:14:17.403Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: aborting pipeline replication: peer="{Voter server-f5786679-9a7b-64d5-1a72-092063626a84 f5786679-9a7b-64d5-1a72-092063626a84}"
2025-06-26T05:14:17.403Z [ERROR] server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: failed to get log: index=23 error="log not found"
2025-06-26T05:14:17.403Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: aborting pipeline replication: peer="{Voter server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc e5eccde2-6b1b-0030-bbee-a8e7626df4bc}"
2025-06-26T05:14:17.403Z [ERROR] server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: failed to get log: index=23 error="log not found"
2025-06-26T05:14:17.403Z [INFO]  creating new snapshot: path=/tmp/raft2583187891/snapshots/2-10013-1750914857403.tmp
2025-06-26T05:14:17.403Z [INFO]  server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:17.403Z [INFO]  server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc: copied to local snapshot: bytes=71
2025-06-26T05:14:17.403Z [INFO]  server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc: snapshot restore progress: id=2-10013-1750914857403 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:17.403Z [INFO]  server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc: Installed remote snapshot
2025-06-26T05:14:17.403Z [WARN]  server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc: skipping application of old log: index=22
2025-06-26T05:14:17.403Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: pipelining replication: peer="{Voter server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc e5eccde2-6b1b-0030-bbee-a8e7626df4bc}"
2025-06-26T05:14:17.403Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:17.413Z [ERROR] server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: failed to get log: index=23 error="log not found"
2025-06-26T05:14:17.413Z [INFO]  creating new snapshot: path=/tmp/raft1487981083/snapshots/2-10013-1750914857413.tmp
2025-06-26T05:14:17.414Z [INFO]  server-f5786679-9a7b-64d5-1a72-092063626a84: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:17.414Z [INFO]  server-f5786679-9a7b-64d5-1a72-092063626a84: copied to local snapshot: bytes=71
2025-06-26T05:14:17.414Z [INFO]  server-f5786679-9a7b-64d5-1a72-092063626a84: snapshot restore progress: id=2-10013-1750914857413 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:17.414Z [INFO]  server-f5786679-9a7b-64d5-1a72-092063626a84: Installed remote snapshot
2025-06-26T05:14:17.414Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: pipelining replication: peer="{Voter server-f5786679-9a7b-64d5-1a72-092063626a84 f5786679-9a7b-64d5-1a72-092063626a84}"
2025-06-26T05:14:18.409Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:17.403846283 +0000 UTC m=+85.552956561 (2 nodes), 600ns from start of poll, 1.58135312s from cluster start. Timeout at 2025-06-26 05:14:18.409606563 +0000 UTC m=+86.558716801, 1.00576024s after stability
2025-06-26T05:14:18.430Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: aborting pipeline replication: peer="{Voter server-e5eccde2-6b1b-0030-bbee-a8e7626df4bc e5eccde2-6b1b-0030-bbee-a8e7626df4bc}"
2025-06-26T05:14:18.430Z [INFO]  server-b1b7da20-e303-20cd-2faa-cc2fe3c2823f: aborting pipeline replication: peer="{Voter server-f5786679-9a7b-64d5-1a72-092063626a84 f5786679-9a7b-64d5-1a72-092063626a84}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2025-06-26T05:14:18.430Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8321d6f5-6233-03b2-b847-553475503682 Address:8321d6f5-6233-03b2-b847-553475503682} {Suffrage:Voter ID:server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3 Address:a0d37144-b8db-bfae-1406-a5c6ecdf6ae3} {Suffrage:Voter ID:server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919 Address:7b1e2e99-4be3-ae3a-6292-b1c91c44f919}]"
2025-06-26T05:14:18.430Z [INFO]  server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8321d6f5-6233-03b2-b847-553475503682 Address:8321d6f5-6233-03b2-b847-553475503682} {Suffrage:Voter ID:server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3 Address:a0d37144-b8db-bfae-1406-a5c6ecdf6ae3} {Suffrage:Voter ID:server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919 Address:7b1e2e99-4be3-ae3a-6292-b1c91c44f919}]"
2025-06-26T05:14:18.430Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8321d6f5-6233-03b2-b847-553475503682 Address:8321d6f5-6233-03b2-b847-553475503682} {Suffrage:Voter ID:server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3 Address:a0d37144-b8db-bfae-1406-a5c6ecdf6ae3} {Suffrage:Voter ID:server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919 Address:7b1e2e99-4be3-ae3a-6292-b1c91c44f919}]"
2025-06-26T05:14:18.430Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:18.430Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: entering follower state: follower="Node at 8321d6f5-6233-03b2-b847-553475503682 [Follower]" leader-address= leader-id=
2025-06-26T05:14:18.431Z [INFO]  server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3: entering follower state: follower="Node at a0d37144-b8db-bfae-1406-a5c6ecdf6ae3 [Follower]" leader-address= leader-id=
2025-06-26T05:14:18.431Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: entering follower state: follower="Node at 7b1e2e99-4be3-ae3a-6292-b1c91c44f919 [Follower]" leader-address= leader-id=
2025-06-26T05:14:19.113Z [WARN]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:19.113Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: entering candidate state: node="Node at 7b1e2e99-4be3-ae3a-6292-b1c91c44f919 [Candidate]" term=2
2025-06-26T05:14:19.113Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: election won: term=2 tally=2
2025-06-26T05:14:19.113Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: entering leader state: leader="Node at 7b1e2e99-4be3-ae3a-6292-b1c91c44f919 [Leader]"
2025-06-26T05:14:19.113Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: added peer, starting replication: peer=server-8321d6f5-6233-03b2-b847-553475503682
2025-06-26T05:14:19.113Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: added peer, starting replication: peer=server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3
2025-06-26T05:14:19.114Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: pipelining replication: peer="{Voter server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3 a0d37144-b8db-bfae-1406-a5c6ecdf6ae3}"
2025-06-26T05:14:19.114Z [WARN]  server-8321d6f5-6233-03b2-b847-553475503682: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:19.114Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: entering candidate state: node="Node at 8321d6f5-6233-03b2-b847-553475503682 [Candidate]" term=2
2025-06-26T05:14:19.114Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: duplicate requestVote for same term: term=2
2025-06-26T05:14:19.114Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: entering follower state: follower="Node at 8321d6f5-6233-03b2-b847-553475503682 [Follower]" leader-address=7b1e2e99-4be3-ae3a-6292-b1c91c44f919 leader-id=server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919
2025-06-26T05:14:19.114Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: pipelining replication: peer="{Voter server-8321d6f5-6233-03b2-b847-553475503682 8321d6f5-6233-03b2-b847-553475503682}"
2025-06-26T05:14:19.114Z [WARN]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: rejecting vote request since we have a leader: from=8321d6f5-6233-03b2-b847-553475503682 leader=7b1e2e99-4be3-ae3a-6292-b1c91c44f919 leader-id=server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919
2025-06-26T05:14:19.114Z [WARN]  server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3: rejecting vote request since we have a leader: from=8321d6f5-6233-03b2-b847-553475503682 leader=7b1e2e99-4be3-ae3a-6292-b1c91c44f919 leader-id=server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919
2025-06-26T05:14:20.122Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:19.114270603 +0000 UTC m=+87.263380881 (1 nodes), 683.30712ms from start of poll, 683.51732ms from cluster start. Timeout at 2025-06-26 05:14:20.122064403 +0000 UTC m=+88.271174681, 1.0077938s after stability
2025-06-26T05:14:20.122Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: starting snapshot up to: index=12
2025-06-26T05:14:20.122Z [INFO]  creating new snapshot: path=/tmp/raft2826670361/snapshots/2-12-1750914860122.tmp
2025-06-26T05:14:20.122Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: snapshot complete up to: index=12
2025-06-26T05:14:20.123Z [INFO]  creating new snapshot: path=/tmp/raft2826670361/snapshots/2-10013-1750914860123.tmp
2025-06-26T05:14:20.123Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: copied to local snapshot: bytes=71
2025-06-26T05:14:20.123Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: snapshot restore progress: id=2-10013-1750914860123 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:20.123Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: removing all old logs from log store
2025-06-26T05:14:20.123Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: compacting logs: from=1 to=22
2025-06-26T05:14:20.123Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: restored user snapshot: index=10013
2025-06-26T05:14:20.123Z [ERROR] server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: failed to get log: index=22 error="log not found"
2025-06-26T05:14:20.123Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: aborting pipeline replication: peer="{Voter server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3 a0d37144-b8db-bfae-1406-a5c6ecdf6ae3}"
2025-06-26T05:14:20.123Z [ERROR] server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: failed to get log: index=22 error="log not found"
2025-06-26T05:14:20.123Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: aborting pipeline replication: peer="{Voter server-8321d6f5-6233-03b2-b847-553475503682 8321d6f5-6233-03b2-b847-553475503682}"
2025-06-26T05:14:20.129Z [ERROR] server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: failed to get log: index=22 error="log not found"
2025-06-26T05:14:20.130Z [INFO]  creating new snapshot: path=/tmp/raft3758097102/snapshots/2-10013-1750914860130.tmp
2025-06-26T05:14:20.130Z [INFO]  server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:20.130Z [INFO]  server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3: copied to local snapshot: bytes=71
2025-06-26T05:14:20.130Z [INFO]  server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3: snapshot restore progress: id=2-10013-1750914860130 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:20.130Z [INFO]  server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3: removing all old logs from log store
2025-06-26T05:14:20.130Z [INFO]  server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3: compacting logs: from=1 to=22
2025-06-26T05:14:20.130Z [INFO]  server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3: Installed remote snapshot
2025-06-26T05:14:20.130Z [WARN]  server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3: skipping application of old log: index=22
2025-06-26T05:14:20.130Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: pipelining replication: peer="{Voter server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3 a0d37144-b8db-bfae-1406-a5c6ecdf6ae3}"
2025-06-26T05:14:20.130Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:20.137Z [ERROR] server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: failed to get log: index=22 error="log not found"
2025-06-26T05:14:20.138Z [INFO]  creating new snapshot: path=/tmp/raft1788935184/snapshots/2-10013-1750914860138.tmp
2025-06-26T05:14:20.138Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:20.138Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: copied to local snapshot: bytes=71
2025-06-26T05:14:20.138Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: snapshot restore progress: id=2-10013-1750914860138 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:20.138Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: removing all old logs from log store
2025-06-26T05:14:20.138Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: compacting logs: from=1 to=22
2025-06-26T05:14:20.138Z [INFO]  server-8321d6f5-6233-03b2-b847-553475503682: Installed remote snapshot
2025-06-26T05:14:20.138Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: pipelining replication: peer="{Voter server-8321d6f5-6233-03b2-b847-553475503682 8321d6f5-6233-03b2-b847-553475503682}"
2025-06-26T05:14:21.137Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:20.130741923 +0000 UTC m=+88.279852161 (2 nodes), 560ns from start of poll, 1.6999886s from cluster start. Timeout at 2025-06-26 05:14:21.137944203 +0000 UTC m=+89.287054481, 1.00720232s after stability
2025-06-26T05:14:21.154Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: aborting pipeline replication: peer="{Voter server-a0d37144-b8db-bfae-1406-a5c6ecdf6ae3 a0d37144-b8db-bfae-1406-a5c6ecdf6ae3}"
2025-06-26T05:14:21.154Z [INFO]  server-7b1e2e99-4be3-ae3a-6292-b1c91c44f919: aborting pipeline replication: peer="{Voter server-8321d6f5-6233-03b2-b847-553475503682 8321d6f5-6233-03b2-b847-553475503682}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2025-06-26T05:14:21.155Z [INFO]  server-f579519b-302e-e930-47ab-ff6db269fe04: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f579519b-302e-e930-47ab-ff6db269fe04 Address:f579519b-302e-e930-47ab-ff6db269fe04} {Suffrage:Voter ID:server-a73b4c74-bc6b-333e-1536-4042976e1e21 Address:a73b4c74-bc6b-333e-1536-4042976e1e21} {Suffrage:Voter ID:server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4 Address:e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4}]"
2025-06-26T05:14:21.155Z [INFO]  server-a73b4c74-bc6b-333e-1536-4042976e1e21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f579519b-302e-e930-47ab-ff6db269fe04 Address:f579519b-302e-e930-47ab-ff6db269fe04} {Suffrage:Voter ID:server-a73b4c74-bc6b-333e-1536-4042976e1e21 Address:a73b4c74-bc6b-333e-1536-4042976e1e21} {Suffrage:Voter ID:server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4 Address:e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4}]"
2025-06-26T05:14:21.155Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f579519b-302e-e930-47ab-ff6db269fe04 Address:f579519b-302e-e930-47ab-ff6db269fe04} {Suffrage:Voter ID:server-a73b4c74-bc6b-333e-1536-4042976e1e21 Address:a73b4c74-bc6b-333e-1536-4042976e1e21} {Suffrage:Voter ID:server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4 Address:e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4}]"
2025-06-26T05:14:21.155Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:21.158Z [INFO]  server-a73b4c74-bc6b-333e-1536-4042976e1e21: entering follower state: follower="Node at a73b4c74-bc6b-333e-1536-4042976e1e21 [Follower]" leader-address= leader-id=
2025-06-26T05:14:21.158Z [INFO]  server-f579519b-302e-e930-47ab-ff6db269fe04: entering follower state: follower="Node at f579519b-302e-e930-47ab-ff6db269fe04 [Follower]" leader-address= leader-id=
2025-06-26T05:14:21.158Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: entering follower state: follower="Node at e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4 [Follower]" leader-address= leader-id=
2025-06-26T05:14:21.729Z [WARN]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:21.729Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: entering candidate state: node="Node at e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4 [Candidate]" term=2
2025-06-26T05:14:21.729Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: election won: term=2 tally=2
2025-06-26T05:14:21.729Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: entering leader state: leader="Node at e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4 [Leader]"
2025-06-26T05:14:21.729Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: added peer, starting replication: peer=server-f579519b-302e-e930-47ab-ff6db269fe04
2025-06-26T05:14:21.729Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: added peer, starting replication: peer=server-a73b4c74-bc6b-333e-1536-4042976e1e21
2025-06-26T05:14:21.730Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: pipelining replication: peer="{Voter server-a73b4c74-bc6b-333e-1536-4042976e1e21 a73b4c74-bc6b-333e-1536-4042976e1e21}"
2025-06-26T05:14:21.730Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: pipelining replication: peer="{Voter server-f579519b-302e-e930-47ab-ff6db269fe04 f579519b-302e-e930-47ab-ff6db269fe04}"
2025-06-26T05:14:22.737Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:21.730210083 +0000 UTC m=+89.879320361 (1 nodes), 574.49612ms from start of poll, 575.1218ms from cluster start. Timeout at 2025-06-26 05:14:22.737613683 +0000 UTC m=+90.886723961, 1.0074036s after stability
2025-06-26T05:14:22.742Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: starting snapshot up to: index=12
2025-06-26T05:14:22.742Z [INFO]  creating new snapshot: path=/tmp/raft3398394981/snapshots/2-12-1750914862742.tmp
2025-06-26T05:14:22.742Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: snapshot complete up to: index=12
2025-06-26T05:14:22.743Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90d61ce3-49df-cb9c-bfb0-b436810289b8 Address:90d61ce3-49df-cb9c-bfb0-b436810289b8} {Suffrage:Voter ID:server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c Address:6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c} {Suffrage:Voter ID:server-2744c3e4-fbc9-a926-dbc6-35034ef0aece Address:2744c3e4-fbc9-a926-dbc6-35034ef0aece}]"
2025-06-26T05:14:22.743Z [INFO]  server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90d61ce3-49df-cb9c-bfb0-b436810289b8 Address:90d61ce3-49df-cb9c-bfb0-b436810289b8} {Suffrage:Voter ID:server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c Address:6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c} {Suffrage:Voter ID:server-2744c3e4-fbc9-a926-dbc6-35034ef0aece Address:2744c3e4-fbc9-a926-dbc6-35034ef0aece}]"
2025-06-26T05:14:22.743Z [INFO]  server-2744c3e4-fbc9-a926-dbc6-35034ef0aece: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90d61ce3-49df-cb9c-bfb0-b436810289b8 Address:90d61ce3-49df-cb9c-bfb0-b436810289b8} {Suffrage:Voter ID:server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c Address:6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c} {Suffrage:Voter ID:server-2744c3e4-fbc9-a926-dbc6-35034ef0aece Address:2744c3e4-fbc9-a926-dbc6-35034ef0aece}]"
2025-06-26T05:14:22.743Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:22.743Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: entering follower state: follower="Node at 90d61ce3-49df-cb9c-bfb0-b436810289b8 [Follower]" leader-address= leader-id=
2025-06-26T05:14:22.743Z [INFO]  server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c: entering follower state: follower="Node at 6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c [Follower]" leader-address= leader-id=
2025-06-26T05:14:22.743Z [INFO]  server-2744c3e4-fbc9-a926-dbc6-35034ef0aece: entering follower state: follower="Node at 2744c3e4-fbc9-a926-dbc6-35034ef0aece [Follower]" leader-address= leader-id=
2025-06-26T05:14:23.457Z [WARN]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:23.457Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: entering candidate state: node="Node at 90d61ce3-49df-cb9c-bfb0-b436810289b8 [Candidate]" term=2
2025-06-26T05:14:23.458Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: election won: term=2 tally=2
2025-06-26T05:14:23.458Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: entering leader state: leader="Node at 90d61ce3-49df-cb9c-bfb0-b436810289b8 [Leader]"
2025-06-26T05:14:23.458Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: added peer, starting replication: peer=server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c
2025-06-26T05:14:23.458Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: added peer, starting replication: peer=server-2744c3e4-fbc9-a926-dbc6-35034ef0aece
2025-06-26T05:14:23.458Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: pipelining replication: peer="{Voter server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c 6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c}"
2025-06-26T05:14:23.458Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: pipelining replication: peer="{Voter server-2744c3e4-fbc9-a926-dbc6-35034ef0aece 2744c3e4-fbc9-a926-dbc6-35034ef0aece}"
2025-06-26T05:14:24.469Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:23.458069443 +0000 UTC m=+91.607179681 (1 nodes), 714.81696ms from start of poll, 715.03708ms from cluster start. Timeout at 2025-06-26 05:14:24.469685723 +0000 UTC m=+92.618796001, 1.01161632s after stability
2025-06-26T05:14:24.469Z [INFO]  creating new snapshot: path=/tmp/raft3052318008/snapshots/2-10013-1750914864469.tmp
2025-06-26T05:14:24.470Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: copied to local snapshot: bytes=71
2025-06-26T05:14:24.470Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: snapshot restore progress: id=2-10013-1750914864469 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:24.470Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: restored user snapshot: index=10013
2025-06-26T05:14:24.470Z [ERROR] server-90d61ce3-49df-cb9c-bfb0-b436810289b8: failed to get log: index=3 error="log not found"
2025-06-26T05:14:24.470Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: aborting pipeline replication: peer="{Voter server-2744c3e4-fbc9-a926-dbc6-35034ef0aece 2744c3e4-fbc9-a926-dbc6-35034ef0aece}"
2025-06-26T05:14:24.470Z [ERROR] server-90d61ce3-49df-cb9c-bfb0-b436810289b8: failed to get log: index=3 error="log not found"
2025-06-26T05:14:24.470Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: aborting pipeline replication: peer="{Voter server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c 6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c}"
2025-06-26T05:14:24.477Z [ERROR] server-90d61ce3-49df-cb9c-bfb0-b436810289b8: failed to get log: index=3 error="log not found"
2025-06-26T05:14:24.478Z [INFO]  creating new snapshot: path=/tmp/raft1025966304/snapshots/2-10013-1750914864478.tmp
2025-06-26T05:14:24.478Z [INFO]  server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:24.478Z [INFO]  server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c: copied to local snapshot: bytes=71
2025-06-26T05:14:24.478Z [INFO]  server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c: snapshot restore progress: id=2-10013-1750914864478 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:24.478Z [INFO]  server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c: Installed remote snapshot
2025-06-26T05:14:24.478Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: pipelining replication: peer="{Voter server-6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c 6a6bcb9f-d382-11e7-cd7e-eed8e5d3716c}"
2025-06-26T05:14:24.478Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:24.478Z [ERROR] server-90d61ce3-49df-cb9c-bfb0-b436810289b8: failed to get log: index=3 error="log not found"
2025-06-26T05:14:24.478Z [INFO]  creating new snapshot: path=/tmp/raft3382557951/snapshots/2-10013-1750914864478.tmp
2025-06-26T05:14:24.479Z [INFO]  server-2744c3e4-fbc9-a926-dbc6-35034ef0aece: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:24.479Z [INFO]  server-2744c3e4-fbc9-a926-dbc6-35034ef0aece: copied to local snapshot: bytes=71
2025-06-26T05:14:24.479Z [INFO]  server-2744c3e4-fbc9-a926-dbc6-35034ef0aece: snapshot restore progress: id=2-10013-1750914864478 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:24.479Z [INFO]  server-2744c3e4-fbc9-a926-dbc6-35034ef0aece: Installed remote snapshot
2025-06-26T05:14:24.479Z [INFO]  server-90d61ce3-49df-cb9c-bfb0-b436810289b8: pipelining replication: peer="{Voter server-2744c3e4-fbc9-a926-dbc6-35034ef0aece 2744c3e4-fbc9-a926-dbc6-35034ef0aece}"
2025-06-26T05:14:25.485Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:24.478711803 +0000 UTC m=+92.627822081 (2 nodes), 560ns from start of poll, 1.73567948s from cluster start. Timeout at 2025-06-26 05:14:25.485825963 +0000 UTC m=+93.634936241, 1.00711416s after stability
2025-06-26T05:14:25.502Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: aborting pipeline replication: peer="{Voter server-a73b4c74-bc6b-333e-1536-4042976e1e21 a73b4c74-bc6b-333e-1536-4042976e1e21}"
2025-06-26T05:14:25.502Z [INFO]  server-e2bb2ecf-57e2-bdcc-bf2b-6a1cebed81c4: aborting pipeline replication: peer="{Voter server-f579519b-302e-e930-47ab-ff6db269fe04 f579519b-302e-e930-47ab-ff6db269fe04}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2025-06-26T05:14:25.502Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-045200db-8f68-9456-a786-acc978396f82 Address:045200db-8f68-9456-a786-acc978396f82} {Suffrage:Voter ID:server-93523630-7699-ef83-19da-1cd8e9f885ce Address:93523630-7699-ef83-19da-1cd8e9f885ce} {Suffrage:Voter ID:server-ea96c0a3-be68-bd98-e8d4-b039408da6e0 Address:ea96c0a3-be68-bd98-e8d4-b039408da6e0}]"
2025-06-26T05:14:25.502Z [INFO]  server-93523630-7699-ef83-19da-1cd8e9f885ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-045200db-8f68-9456-a786-acc978396f82 Address:045200db-8f68-9456-a786-acc978396f82} {Suffrage:Voter ID:server-93523630-7699-ef83-19da-1cd8e9f885ce Address:93523630-7699-ef83-19da-1cd8e9f885ce} {Suffrage:Voter ID:server-ea96c0a3-be68-bd98-e8d4-b039408da6e0 Address:ea96c0a3-be68-bd98-e8d4-b039408da6e0}]"
2025-06-26T05:14:25.502Z [INFO]  server-ea96c0a3-be68-bd98-e8d4-b039408da6e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-045200db-8f68-9456-a786-acc978396f82 Address:045200db-8f68-9456-a786-acc978396f82} {Suffrage:Voter ID:server-93523630-7699-ef83-19da-1cd8e9f885ce Address:93523630-7699-ef83-19da-1cd8e9f885ce} {Suffrage:Voter ID:server-ea96c0a3-be68-bd98-e8d4-b039408da6e0 Address:ea96c0a3-be68-bd98-e8d4-b039408da6e0}]"
2025-06-26T05:14:25.502Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:25.502Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: entering follower state: follower="Node at 045200db-8f68-9456-a786-acc978396f82 [Follower]" leader-address= leader-id=
2025-06-26T05:14:25.503Z [INFO]  server-93523630-7699-ef83-19da-1cd8e9f885ce: entering follower state: follower="Node at 93523630-7699-ef83-19da-1cd8e9f885ce [Follower]" leader-address= leader-id=
2025-06-26T05:14:25.503Z [INFO]  server-ea96c0a3-be68-bd98-e8d4-b039408da6e0: entering follower state: follower="Node at ea96c0a3-be68-bd98-e8d4-b039408da6e0 [Follower]" leader-address= leader-id=
2025-06-26T05:14:26.045Z [WARN]  server-045200db-8f68-9456-a786-acc978396f82: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:26.045Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: entering candidate state: node="Node at 045200db-8f68-9456-a786-acc978396f82 [Candidate]" term=2
2025-06-26T05:14:26.045Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: election won: term=2 tally=2
2025-06-26T05:14:26.045Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: entering leader state: leader="Node at 045200db-8f68-9456-a786-acc978396f82 [Leader]"
2025-06-26T05:14:26.045Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: added peer, starting replication: peer=server-93523630-7699-ef83-19da-1cd8e9f885ce
2025-06-26T05:14:26.045Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: added peer, starting replication: peer=server-ea96c0a3-be68-bd98-e8d4-b039408da6e0
2025-06-26T05:14:26.045Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: pipelining replication: peer="{Voter server-ea96c0a3-be68-bd98-e8d4-b039408da6e0 ea96c0a3-be68-bd98-e8d4-b039408da6e0}"
2025-06-26T05:14:26.046Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: pipelining replication: peer="{Voter server-93523630-7699-ef83-19da-1cd8e9f885ce 93523630-7699-ef83-19da-1cd8e9f885ce}"
2025-06-26T05:14:27.053Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:26.046335323 +0000 UTC m=+94.195445601 (1 nodes), 543.40252ms from start of poll, 543.63452ms from cluster start. Timeout at 2025-06-26 05:14:27.053609683 +0000 UTC m=+95.202719961, 1.00727436s after stability
2025-06-26T05:14:27.054Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: starting snapshot up to: index=12
2025-06-26T05:14:27.054Z [INFO]  creating new snapshot: path=/tmp/raft1471752454/snapshots/2-12-1750914867054.tmp
2025-06-26T05:14:27.054Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: snapshot complete up to: index=12
2025-06-26T05:14:27.055Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e2fd71b-3e45-0235-8f37-f794b580b8d3 Address:2e2fd71b-3e45-0235-8f37-f794b580b8d3} {Suffrage:Voter ID:server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f Address:f7ba19be-4c3e-0f88-3f39-c3b27d193e5f} {Suffrage:Voter ID:server-8898074b-af26-d5ce-d445-8d5f5fc37ebe Address:8898074b-af26-d5ce-d445-8d5f5fc37ebe}]"
2025-06-26T05:14:27.055Z [INFO]  server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e2fd71b-3e45-0235-8f37-f794b580b8d3 Address:2e2fd71b-3e45-0235-8f37-f794b580b8d3} {Suffrage:Voter ID:server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f Address:f7ba19be-4c3e-0f88-3f39-c3b27d193e5f} {Suffrage:Voter ID:server-8898074b-af26-d5ce-d445-8d5f5fc37ebe Address:8898074b-af26-d5ce-d445-8d5f5fc37ebe}]"
2025-06-26T05:14:27.055Z [INFO]  server-8898074b-af26-d5ce-d445-8d5f5fc37ebe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e2fd71b-3e45-0235-8f37-f794b580b8d3 Address:2e2fd71b-3e45-0235-8f37-f794b580b8d3} {Suffrage:Voter ID:server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f Address:f7ba19be-4c3e-0f88-3f39-c3b27d193e5f} {Suffrage:Voter ID:server-8898074b-af26-d5ce-d445-8d5f5fc37ebe Address:8898074b-af26-d5ce-d445-8d5f5fc37ebe}]"
2025-06-26T05:14:27.055Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:27.055Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: entering follower state: follower="Node at 2e2fd71b-3e45-0235-8f37-f794b580b8d3 [Follower]" leader-address= leader-id=
2025-06-26T05:14:27.055Z [INFO]  server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f: entering follower state: follower="Node at f7ba19be-4c3e-0f88-3f39-c3b27d193e5f [Follower]" leader-address= leader-id=
2025-06-26T05:14:27.055Z [INFO]  server-8898074b-af26-d5ce-d445-8d5f5fc37ebe: entering follower state: follower="Node at 8898074b-af26-d5ce-d445-8d5f5fc37ebe [Follower]" leader-address= leader-id=
2025-06-26T05:14:27.762Z [WARN]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:27.762Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: entering candidate state: node="Node at 2e2fd71b-3e45-0235-8f37-f794b580b8d3 [Candidate]" term=2
2025-06-26T05:14:27.762Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: election won: term=2 tally=2
2025-06-26T05:14:27.762Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: entering leader state: leader="Node at 2e2fd71b-3e45-0235-8f37-f794b580b8d3 [Leader]"
2025-06-26T05:14:27.762Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: added peer, starting replication: peer=server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f
2025-06-26T05:14:27.762Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: added peer, starting replication: peer=server-8898074b-af26-d5ce-d445-8d5f5fc37ebe
2025-06-26T05:14:27.762Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: pipelining replication: peer="{Voter server-8898074b-af26-d5ce-d445-8d5f5fc37ebe 8898074b-af26-d5ce-d445-8d5f5fc37ebe}"
2025-06-26T05:14:27.762Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: pipelining replication: peer="{Voter server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f f7ba19be-4c3e-0f88-3f39-c3b27d193e5f}"
2025-06-26T05:14:28.769Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:27.763042283 +0000 UTC m=+95.912152561 (1 nodes), 707.67248ms from start of poll, 707.87548ms from cluster start. Timeout at 2025-06-26 05:14:28.769628803 +0000 UTC m=+96.918739041, 1.00658648s after stability
2025-06-26T05:14:28.769Z [INFO]  creating new snapshot: path=/tmp/raft1497829251/snapshots/2-10013-1750914868769.tmp
2025-06-26T05:14:28.770Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: copied to local snapshot: bytes=71
2025-06-26T05:14:28.770Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: snapshot restore progress: id=2-10013-1750914868769 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:28.770Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: removing all old logs from log store
2025-06-26T05:14:28.770Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: compacting logs: from=1 to=2
2025-06-26T05:14:28.770Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: restored user snapshot: index=10013
2025-06-26T05:14:28.770Z [ERROR] server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: failed to get log: index=2 error="log not found"
2025-06-26T05:14:28.770Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: aborting pipeline replication: peer="{Voter server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f f7ba19be-4c3e-0f88-3f39-c3b27d193e5f}"
2025-06-26T05:14:28.770Z [ERROR] server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: failed to get log: index=2 error="log not found"
2025-06-26T05:14:28.770Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: aborting pipeline replication: peer="{Voter server-8898074b-af26-d5ce-d445-8d5f5fc37ebe 8898074b-af26-d5ce-d445-8d5f5fc37ebe}"
2025-06-26T05:14:28.770Z [ERROR] server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: failed to get log: index=2 error="log not found"
2025-06-26T05:14:28.770Z [INFO]  creating new snapshot: path=/tmp/raft108508546/snapshots/2-10013-1750914868770.tmp
2025-06-26T05:14:28.770Z [INFO]  server-8898074b-af26-d5ce-d445-8d5f5fc37ebe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:28.770Z [INFO]  server-8898074b-af26-d5ce-d445-8d5f5fc37ebe: copied to local snapshot: bytes=71
2025-06-26T05:14:28.770Z [INFO]  server-8898074b-af26-d5ce-d445-8d5f5fc37ebe: snapshot restore progress: id=2-10013-1750914868770 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:28.770Z [INFO]  server-8898074b-af26-d5ce-d445-8d5f5fc37ebe: removing all old logs from log store
2025-06-26T05:14:28.770Z [INFO]  server-8898074b-af26-d5ce-d445-8d5f5fc37ebe: compacting logs: from=1 to=2
2025-06-26T05:14:28.770Z [INFO]  server-8898074b-af26-d5ce-d445-8d5f5fc37ebe: Installed remote snapshot
2025-06-26T05:14:28.771Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: pipelining replication: peer="{Voter server-8898074b-af26-d5ce-d445-8d5f5fc37ebe 8898074b-af26-d5ce-d445-8d5f5fc37ebe}"
2025-06-26T05:14:28.771Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:28.778Z [ERROR] server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: failed to get log: index=2 error="log not found"
2025-06-26T05:14:28.778Z [INFO]  creating new snapshot: path=/tmp/raft2026396019/snapshots/2-10013-1750914868778.tmp
2025-06-26T05:14:28.778Z [INFO]  server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:28.778Z [INFO]  server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f: copied to local snapshot: bytes=71
2025-06-26T05:14:28.779Z [INFO]  server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f: snapshot restore progress: id=2-10013-1750914868778 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-06-26T05:14:28.779Z [INFO]  server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f: removing all old logs from log store
2025-06-26T05:14:28.779Z [INFO]  server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f: compacting logs: from=1 to=2
2025-06-26T05:14:28.779Z [INFO]  server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f: Installed remote snapshot
2025-06-26T05:14:28.779Z [INFO]  server-2e2fd71b-3e45-0235-8f37-f794b580b8d3: pipelining replication: peer="{Voter server-f7ba19be-4c3e-0f88-3f39-c3b27d193e5f f7ba19be-4c3e-0f88-3f39-c3b27d193e5f}"
2025-06-26T05:14:29.777Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:28.771143483 +0000 UTC m=+96.920253721 (2 nodes), 560ns from start of poll, 1.71597664s from cluster start. Timeout at 2025-06-26 05:14:29.777625203 +0000 UTC m=+97.926735441, 1.00648172s after stability
2025-06-26T05:14:29.793Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: aborting pipeline replication: peer="{Voter server-ea96c0a3-be68-bd98-e8d4-b039408da6e0 ea96c0a3-be68-bd98-e8d4-b039408da6e0}"
2025-06-26T05:14:29.794Z [INFO]  server-045200db-8f68-9456-a786-acc978396f82: aborting pipeline replication: peer="{Voter server-93523630-7699-ef83-19da-1cd8e9f885ce 93523630-7699-ef83-19da-1cd8e9f885ce}"
--- PASS: TestRaft_UserRestore (83.03s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.59s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.60s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.35s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.30s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.58s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.82s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.24s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.22s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.57s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.64s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.42s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.24s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.59s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.17s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.18s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.59s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.66s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.38s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.30s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.72s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.35s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.29s)
=== RUN   TestRaft_SendSnapshotFollower
2025-06-26T05:14:29.794Z [INFO]  server-8fc6767e-a63b-8525-b41e-924589f22625: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8fc6767e-a63b-8525-b41e-924589f22625 Address:8fc6767e-a63b-8525-b41e-924589f22625} {Suffrage:Voter ID:server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e Address:15f4d69e-fff6-7c69-e72d-1c278c3f4d9e} {Suffrage:Voter ID:server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17 Address:623f18ff-c4c7-abf6-d084-dbb2b39d6c17}]"
2025-06-26T05:14:29.794Z [INFO]  server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8fc6767e-a63b-8525-b41e-924589f22625 Address:8fc6767e-a63b-8525-b41e-924589f22625} {Suffrage:Voter ID:server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e Address:15f4d69e-fff6-7c69-e72d-1c278c3f4d9e} {Suffrage:Voter ID:server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17 Address:623f18ff-c4c7-abf6-d084-dbb2b39d6c17}]"
2025-06-26T05:14:29.794Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8fc6767e-a63b-8525-b41e-924589f22625 Address:8fc6767e-a63b-8525-b41e-924589f22625} {Suffrage:Voter ID:server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e Address:15f4d69e-fff6-7c69-e72d-1c278c3f4d9e} {Suffrage:Voter ID:server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17 Address:623f18ff-c4c7-abf6-d084-dbb2b39d6c17}]"
2025-06-26T05:14:29.794Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:29.794Z [INFO]  server-8fc6767e-a63b-8525-b41e-924589f22625: entering follower state: follower="Node at 8fc6767e-a63b-8525-b41e-924589f22625 [Follower]" leader-address= leader-id=
2025-06-26T05:14:29.794Z [INFO]  server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e: entering follower state: follower="Node at 15f4d69e-fff6-7c69-e72d-1c278c3f4d9e [Follower]" leader-address= leader-id=
2025-06-26T05:14:29.794Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: entering follower state: follower="Node at 623f18ff-c4c7-abf6-d084-dbb2b39d6c17 [Follower]" leader-address= leader-id=
2025-06-26T05:14:29.865Z [WARN]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:29.865Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: entering candidate state: node="Node at 623f18ff-c4c7-abf6-d084-dbb2b39d6c17 [Candidate]" term=2
2025-06-26T05:14:29.865Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: election won: term=2 tally=2
2025-06-26T05:14:29.865Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: entering leader state: leader="Node at 623f18ff-c4c7-abf6-d084-dbb2b39d6c17 [Leader]"
2025-06-26T05:14:29.865Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: added peer, starting replication: peer=server-8fc6767e-a63b-8525-b41e-924589f22625
2025-06-26T05:14:29.865Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: added peer, starting replication: peer=server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e
2025-06-26T05:14:29.866Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: pipelining replication: peer="{Voter server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e 15f4d69e-fff6-7c69-e72d-1c278c3f4d9e}"
2025-06-26T05:14:29.866Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: pipelining replication: peer="{Voter server-8fc6767e-a63b-8525-b41e-924589f22625 8fc6767e-a63b-8525-b41e-924589f22625}"
2025-06-26T05:14:29.973Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:29.866309043 +0000 UTC m=+98.015419321 (2 nodes), 71.65316ms from start of poll, 71.89692ms from cluster start. Timeout at 2025-06-26 05:14:29.973831003 +0000 UTC m=+98.122941241, 107.52192ms after stability
2025-06-26T05:14:29.973Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:30.082Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:29.973966523 +0000 UTC m=+98.123076801 (1 nodes), 2µs from start of poll, 179.5544ms from cluster start. Timeout at 2025-06-26 05:14:30.082103043 +0000 UTC m=+98.231213321, 108.13652ms after stability
2025-06-26T05:14:30.082Z [ERROR] server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: failed to pipeline appendEntries: peer="{Voter server-8fc6767e-a63b-8525-b41e-924589f22625 8fc6767e-a63b-8525-b41e-924589f22625}" error="append pipeline closed"
2025-06-26T05:14:30.082Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: aborting pipeline replication: peer="{Voter server-8fc6767e-a63b-8525-b41e-924589f22625 8fc6767e-a63b-8525-b41e-924589f22625}"
2025-06-26T05:14:30.082Z [ERROR] server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: failed to appendEntries to: peer="{Voter server-8fc6767e-a63b-8525-b41e-924589f22625 8fc6767e-a63b-8525-b41e-924589f22625}" error="failed to connect to peer: 8fc6767e-a63b-8525-b41e-924589f22625"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2025-06-26T05:14:30.084Z [ERROR] server-8fc6767e-a63b-8525-b41e-924589f22625: failed to take snapshot: error="nothing new to snapshot"
2025-06-26T05:14:30.084Z [INFO]  server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e: starting snapshot up to: index=87
2025-06-26T05:14:30.084Z [INFO]  creating new snapshot: path=/tmp/raft2004589720/snapshots/2-87-1750914870084.tmp
2025-06-26T05:14:30.085Z [INFO]  server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e: compacting logs: from=1 to=87
2025-06-26T05:14:30.085Z [INFO]  server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e: snapshot complete up to: index=87
2025-06-26T05:14:30.085Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: starting snapshot up to: index=102
2025-06-26T05:14:30.085Z [INFO]  creating new snapshot: path=/tmp/raft3647503839/snapshots/2-102-1750914870085.tmp
2025-06-26T05:14:30.085Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: compacting logs: from=1 to=92
2025-06-26T05:14:30.085Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: snapshot complete up to: index=102
2025-06-26T05:14:30.093Z [ERROR] server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: failed to get log: index=2 error="log not found"
2025-06-26T05:14:30.094Z [INFO]  creating new snapshot: path=/tmp/raft1264769550/snapshots/2-102-1750914870094.tmp
2025-06-26T05:14:30.094Z [INFO]  server-8fc6767e-a63b-8525-b41e-924589f22625: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-06-26T05:14:30.094Z [INFO]  server-8fc6767e-a63b-8525-b41e-924589f22625: copied to local snapshot: bytes=693
2025-06-26T05:14:30.094Z [INFO]  server-8fc6767e-a63b-8525-b41e-924589f22625: snapshot restore progress: id=2-102-1750914870094 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-06-26T05:14:30.094Z [INFO]  server-8fc6767e-a63b-8525-b41e-924589f22625: Installed remote snapshot
2025-06-26T05:14:30.094Z [WARN]  server-8fc6767e-a63b-8525-b41e-924589f22625: skipping application of old log: index=102
2025-06-26T05:14:30.094Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: pipelining replication: peer="{Voter server-8fc6767e-a63b-8525-b41e-924589f22625 8fc6767e-a63b-8525-b41e-924589f22625}"
2025-06-26T05:14:30.101Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: aborting pipeline replication: peer="{Voter server-15f4d69e-fff6-7c69-e72d-1c278c3f4d9e 15f4d69e-fff6-7c69-e72d-1c278c3f4d9e}"
2025-06-26T05:14:30.102Z [INFO]  server-623f18ff-c4c7-abf6-d084-dbb2b39d6c17: aborting pipeline replication: peer="{Voter server-8fc6767e-a63b-8525-b41e-924589f22625 8fc6767e-a63b-8525-b41e-924589f22625}"
--- PASS: TestRaft_SendSnapshotFollower (0.31s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2025-06-26T05:14:30.102Z [INFO]  server-20679ded-6857-bfa0-802c-a8c6e7878876: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20679ded-6857-bfa0-802c-a8c6e7878876 Address:20679ded-6857-bfa0-802c-a8c6e7878876} {Suffrage:Voter ID:server-43426fe1-2c40-b821-71f5-ca4c68894255 Address:43426fe1-2c40-b821-71f5-ca4c68894255} {Suffrage:Voter ID:server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663 Address:2338f1c6-5fbb-6b29-14a0-b379f1b1d663}]"
2025-06-26T05:14:30.102Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20679ded-6857-bfa0-802c-a8c6e7878876 Address:20679ded-6857-bfa0-802c-a8c6e7878876} {Suffrage:Voter ID:server-43426fe1-2c40-b821-71f5-ca4c68894255 Address:43426fe1-2c40-b821-71f5-ca4c68894255} {Suffrage:Voter ID:server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663 Address:2338f1c6-5fbb-6b29-14a0-b379f1b1d663}]"
2025-06-26T05:14:30.102Z [INFO]  server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20679ded-6857-bfa0-802c-a8c6e7878876 Address:20679ded-6857-bfa0-802c-a8c6e7878876} {Suffrage:Voter ID:server-43426fe1-2c40-b821-71f5-ca4c68894255 Address:43426fe1-2c40-b821-71f5-ca4c68894255} {Suffrage:Voter ID:server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663 Address:2338f1c6-5fbb-6b29-14a0-b379f1b1d663}]"
2025-06-26T05:14:30.102Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:30.102Z [INFO]  server-20679ded-6857-bfa0-802c-a8c6e7878876: entering follower state: follower="Node at 20679ded-6857-bfa0-802c-a8c6e7878876 [Follower]" leader-address= leader-id=
2025-06-26T05:14:30.102Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: entering follower state: follower="Node at 43426fe1-2c40-b821-71f5-ca4c68894255 [Follower]" leader-address= leader-id=
2025-06-26T05:14:30.102Z [INFO]  server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663: entering follower state: follower="Node at 2338f1c6-5fbb-6b29-14a0-b379f1b1d663 [Follower]" leader-address= leader-id=
2025-06-26T05:14:30.162Z [WARN]  server-43426fe1-2c40-b821-71f5-ca4c68894255: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:30.162Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: entering candidate state: node="Node at 43426fe1-2c40-b821-71f5-ca4c68894255 [Candidate]" term=2
2025-06-26T05:14:30.162Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: election won: term=2 tally=2
2025-06-26T05:14:30.162Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: entering leader state: leader="Node at 43426fe1-2c40-b821-71f5-ca4c68894255 [Leader]"
2025-06-26T05:14:30.162Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: added peer, starting replication: peer=server-20679ded-6857-bfa0-802c-a8c6e7878876
2025-06-26T05:14:30.162Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: added peer, starting replication: peer=server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663
2025-06-26T05:14:30.162Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: pipelining replication: peer="{Voter server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663 2338f1c6-5fbb-6b29-14a0-b379f1b1d663}"
2025-06-26T05:14:30.162Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: pipelining replication: peer="{Voter server-20679ded-6857-bfa0-802c-a8c6e7878876 20679ded-6857-bfa0-802c-a8c6e7878876}"
2025-06-26T05:14:30.270Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:30.162278523 +0000 UTC m=+98.311388801 (2 nodes), 59.5136ms from start of poll, 59.7574ms from cluster start. Timeout at 2025-06-26 05:14:30.270042043 +0000 UTC m=+98.419152321, 107.76352ms after stability
2025-06-26T05:14:30.270Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:30.377Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:30.270120163 +0000 UTC m=+98.419230441 (1 nodes), 1.88µs from start of poll, 167.59904ms from cluster start. Timeout at 2025-06-26 05:14:30.377773283 +0000 UTC m=+98.526883561, 107.65312ms after stability
2025-06-26T05:14:30.378Z [ERROR] server-43426fe1-2c40-b821-71f5-ca4c68894255: failed to pipeline appendEntries: peer="{Voter server-20679ded-6857-bfa0-802c-a8c6e7878876 20679ded-6857-bfa0-802c-a8c6e7878876}" error="append pipeline closed"
2025-06-26T05:14:30.378Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: aborting pipeline replication: peer="{Voter server-20679ded-6857-bfa0-802c-a8c6e7878876 20679ded-6857-bfa0-802c-a8c6e7878876}"
2025-06-26T05:14:30.378Z [ERROR] server-43426fe1-2c40-b821-71f5-ca4c68894255: failed to appendEntries to: peer="{Voter server-20679ded-6857-bfa0-802c-a8c6e7878876 20679ded-6857-bfa0-802c-a8c6e7878876}" error="failed to connect to peer: 20679ded-6857-bfa0-802c-a8c6e7878876"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2025-06-26T05:14:30.379Z [ERROR] server-20679ded-6857-bfa0-802c-a8c6e7878876: failed to take snapshot: error="nothing new to snapshot"
2025-06-26T05:14:30.379Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: starting snapshot up to: index=102
2025-06-26T05:14:30.379Z [INFO]  creating new snapshot: path=/tmp/raft27237396/snapshots/2-102-1750914870379.tmp
2025-06-26T05:14:30.380Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: compacting logs: from=1 to=92
2025-06-26T05:14:30.380Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: snapshot complete up to: index=102
2025-06-26T05:14:30.380Z [INFO]  server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663: starting snapshot up to: index=44
2025-06-26T05:14:30.380Z [INFO]  creating new snapshot: path=/tmp/raft3663255422/snapshots/2-44-1750914870380.tmp
2025-06-26T05:14:30.380Z [INFO]  server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663: compacting logs: from=1 to=44
2025-06-26T05:14:30.380Z [INFO]  server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663: snapshot complete up to: index=44
2025-06-26T05:14:30.380Z [ERROR] server-43426fe1-2c40-b821-71f5-ca4c68894255: failed to heartbeat to: peer=20679ded-6857-bfa0-802c-a8c6e7878876 backoff time=10ms error="failed to connect to peer: 20679ded-6857-bfa0-802c-a8c6e7878876"
    raft_test.go:1690: [INFO] Finished apply without behind follower
2025-06-26T05:14:30.405Z [ERROR] server-43426fe1-2c40-b821-71f5-ca4c68894255: failed to get log: index=2 error="log not found"
2025-06-26T05:14:30.406Z [INFO]  creating new snapshot: path=/tmp/raft3775403175/snapshots/2-102-1750914870406.tmp
2025-06-26T05:14:30.406Z [INFO]  server-20679ded-6857-bfa0-802c-a8c6e7878876: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-06-26T05:14:30.406Z [INFO]  server-20679ded-6857-bfa0-802c-a8c6e7878876: copied to local snapshot: bytes=693
2025-06-26T05:14:30.406Z [INFO]  server-20679ded-6857-bfa0-802c-a8c6e7878876: snapshot restore progress: id=2-102-1750914870406 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-06-26T05:14:30.406Z [INFO]  server-20679ded-6857-bfa0-802c-a8c6e7878876: Installed remote snapshot
2025-06-26T05:14:30.406Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: pipelining replication: peer="{Voter server-20679ded-6857-bfa0-802c-a8c6e7878876 20679ded-6857-bfa0-802c-a8c6e7878876}"
2025-06-26T05:14:30.415Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: aborting pipeline replication: peer="{Voter server-2338f1c6-5fbb-6b29-14a0-b379f1b1d663 2338f1c6-5fbb-6b29-14a0-b379f1b1d663}"
2025-06-26T05:14:30.416Z [INFO]  server-43426fe1-2c40-b821-71f5-ca4c68894255: aborting pipeline replication: peer="{Voter server-20679ded-6857-bfa0-802c-a8c6e7878876 20679ded-6857-bfa0-802c-a8c6e7878876}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.31s)
=== RUN   TestRaft_ReJoinFollower
2025-06-26T05:14:30.416Z [INFO]  server-69bd4302-90a8-3209-e4ae-6fd2abd4a860: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 Address:69bd4302-90a8-3209-e4ae-6fd2abd4a860} {Suffrage:Voter ID:server-62e23ac4-987c-e60e-b260-fedd743974d1 Address:62e23ac4-987c-e60e-b260-fedd743974d1} {Suffrage:Voter ID:server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64 Address:2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64}]"
2025-06-26T05:14:30.416Z [INFO]  server-62e23ac4-987c-e60e-b260-fedd743974d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 Address:69bd4302-90a8-3209-e4ae-6fd2abd4a860} {Suffrage:Voter ID:server-62e23ac4-987c-e60e-b260-fedd743974d1 Address:62e23ac4-987c-e60e-b260-fedd743974d1} {Suffrage:Voter ID:server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64 Address:2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64}]"
2025-06-26T05:14:30.416Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 Address:69bd4302-90a8-3209-e4ae-6fd2abd4a860} {Suffrage:Voter ID:server-62e23ac4-987c-e60e-b260-fedd743974d1 Address:62e23ac4-987c-e60e-b260-fedd743974d1} {Suffrage:Voter ID:server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64 Address:2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64}]"
2025-06-26T05:14:30.416Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:30.416Z [INFO]  server-69bd4302-90a8-3209-e4ae-6fd2abd4a860: entering follower state: follower="Node at 69bd4302-90a8-3209-e4ae-6fd2abd4a860 [Follower]" leader-address= leader-id=
2025-06-26T05:14:30.416Z [INFO]  server-62e23ac4-987c-e60e-b260-fedd743974d1: entering follower state: follower="Node at 62e23ac4-987c-e60e-b260-fedd743974d1 [Follower]" leader-address= leader-id=
2025-06-26T05:14:30.416Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: entering follower state: follower="Node at 2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64 [Follower]" leader-address= leader-id=
2025-06-26T05:14:30.489Z [WARN]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:30.489Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: entering candidate state: node="Node at 2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64 [Candidate]" term=2
2025-06-26T05:14:30.489Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: election won: term=2 tally=2
2025-06-26T05:14:30.489Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: entering leader state: leader="Node at 2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64 [Leader]"
2025-06-26T05:14:30.489Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: added peer, starting replication: peer=server-69bd4302-90a8-3209-e4ae-6fd2abd4a860
2025-06-26T05:14:30.489Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: added peer, starting replication: peer=server-62e23ac4-987c-e60e-b260-fedd743974d1
2025-06-26T05:14:30.489Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: pipelining replication: peer="{Voter server-62e23ac4-987c-e60e-b260-fedd743974d1 62e23ac4-987c-e60e-b260-fedd743974d1}"
2025-06-26T05:14:30.490Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: pipelining replication: peer="{Voter server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 69bd4302-90a8-3209-e4ae-6fd2abd4a860}"
2025-06-26T05:14:30.598Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:30.490104483 +0000 UTC m=+98.639214761 (1 nodes), 73.38388ms from start of poll, 73.59024ms from cluster start. Timeout at 2025-06-26 05:14:30.598251083 +0000 UTC m=+98.747361361, 108.1466ms after stability
2025-06-26T05:14:30.605Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:30.713Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:30.605769643 +0000 UTC m=+98.754879921 (2 nodes), 1.68µs from start of poll, 189.2554ms from cluster start. Timeout at 2025-06-26 05:14:30.713624243 +0000 UTC m=+98.862734521, 107.8546ms after stability
2025-06-26T05:14:30.713Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: updating configuration: command=RemoveServer server-id=server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 server-addr= servers="[{Suffrage:Voter ID:server-62e23ac4-987c-e60e-b260-fedd743974d1 Address:62e23ac4-987c-e60e-b260-fedd743974d1} {Suffrage:Voter ID:server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64 Address:2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64}]"
2025-06-26T05:14:30.713Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: removed peer, stopping replication: peer=server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 last-index=3
2025-06-26T05:14:30.713Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: aborting pipeline replication: peer="{Voter server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 69bd4302-90a8-3209-e4ae-6fd2abd4a860}"
2025-06-26T05:14:30.814Z [WARN]  server-69bd4302-90a8-3209-e4ae-6fd2abd4a860: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-06-26T05:14:30.822Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: updating configuration: command=AddVoter server-id=server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 server-addr=69bd4302-90a8-3209-e4ae-6fd2abd4a860 servers="[{Suffrage:Voter ID:server-62e23ac4-987c-e60e-b260-fedd743974d1 Address:62e23ac4-987c-e60e-b260-fedd743974d1} {Suffrage:Voter ID:server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64 Address:2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64} {Suffrage:Voter ID:server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 Address:69bd4302-90a8-3209-e4ae-6fd2abd4a860}]"
2025-06-26T05:14:30.822Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: added peer, starting replication: peer=server-69bd4302-90a8-3209-e4ae-6fd2abd4a860
2025-06-26T05:14:30.822Z [WARN]  server-69bd4302-90a8-3209-e4ae-6fd2abd4a860: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2025-06-26T05:14:30.822Z [WARN]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: appendEntries rejected, sending older logs: peer="{Voter server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 69bd4302-90a8-3209-e4ae-6fd2abd4a860}" next=4
2025-06-26T05:14:30.822Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: pipelining replication: peer="{Voter server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 69bd4302-90a8-3209-e4ae-6fd2abd4a860}"
2025-06-26T05:14:30.822Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:30.929Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:30.822608443 +0000 UTC m=+98.971718721 (1 nodes), 400ns from start of poll, 406.0942ms from cluster start. Timeout at 2025-06-26 05:14:30.929620163 +0000 UTC m=+99.078730441, 107.01172ms after stability
2025-06-26T05:14:30.929Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: aborting pipeline replication: peer="{Voter server-62e23ac4-987c-e60e-b260-fedd743974d1 62e23ac4-987c-e60e-b260-fedd743974d1}"
2025-06-26T05:14:30.929Z [INFO]  server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: aborting pipeline replication: peer="{Voter server-69bd4302-90a8-3209-e4ae-6fd2abd4a860 69bd4302-90a8-3209-e4ae-6fd2abd4a860}"
2025-06-26T05:14:31.430Z [ERROR] server-2920d9d3-95e6-5d19-f5e4-1d46ccbb0f64: failed to heartbeat to: peer=69bd4302-90a8-3209-e4ae-6fd2abd4a860 backoff time=10ms error="command timed out"
--- PASS: TestRaft_ReJoinFollower (1.01s)
=== RUN   TestRaft_LeaderLeaseExpire
2025-06-26T05:14:31.430Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8dbecc6b-9568-19fa-3084-832b914ec3b0 Address:8dbecc6b-9568-19fa-3084-832b914ec3b0} {Suffrage:Voter ID:server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 Address:3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}]"
2025-06-26T05:14:31.430Z [INFO]  server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8dbecc6b-9568-19fa-3084-832b914ec3b0 Address:8dbecc6b-9568-19fa-3084-832b914ec3b0} {Suffrage:Voter ID:server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 Address:3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}]"
2025-06-26T05:14:31.430Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:31.430Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: entering follower state: follower="Node at 8dbecc6b-9568-19fa-3084-832b914ec3b0 [Follower]" leader-address= leader-id=
2025-06-26T05:14:31.430Z [INFO]  server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: entering follower state: follower="Node at 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 [Follower]" leader-address= leader-id=
2025-06-26T05:14:31.482Z [WARN]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:31.482Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: entering candidate state: node="Node at 8dbecc6b-9568-19fa-3084-832b914ec3b0 [Candidate]" term=2
2025-06-26T05:14:31.482Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: election won: term=2 tally=2
2025-06-26T05:14:31.482Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: entering leader state: leader="Node at 8dbecc6b-9568-19fa-3084-832b914ec3b0 [Leader]"
2025-06-26T05:14:31.482Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: added peer, starting replication: peer=server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6
2025-06-26T05:14:31.482Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: pipelining replication: peer="{Voter server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}"
2025-06-26T05:14:31.589Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:31.482432523 +0000 UTC m=+99.631542801 (1 nodes), 51.71516ms from start of poll, 51.91076ms from cluster start. Timeout at 2025-06-26 05:14:31.589614563 +0000 UTC m=+99.738724841, 107.18204ms after stability
2025-06-26T05:14:31.597Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:31.706Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:31.597881203 +0000 UTC m=+99.746991481 (1 nodes), 1.92µs from start of poll, 167.35944ms from cluster start. Timeout at 2025-06-26 05:14:31.706036603 +0000 UTC m=+99.855146881, 108.1554ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 [Follower]
2025-06-26T05:14:31.709Z [ERROR] server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to heartbeat to: peer=3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 backoff time=10ms error="failed to connect to peer: 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6"
2025-06-26T05:14:31.717Z [ERROR] server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to pipeline appendEntries: peer="{Voter server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}" error="append pipeline closed"
2025-06-26T05:14:31.717Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: aborting pipeline replication: peer="{Voter server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}"
2025-06-26T05:14:31.725Z [ERROR] server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to appendEntries to: peer="{Voter server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}" error="failed to connect to peer: 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6"
2025-06-26T05:14:31.733Z [ERROR] server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to heartbeat to: peer=3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 backoff time=10ms error="failed to connect to peer: 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6"
2025-06-26T05:14:31.745Z [ERROR] server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to appendEntries to: peer="{Voter server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}" error="failed to connect to peer: 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6"
2025-06-26T05:14:31.753Z [ERROR] server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to heartbeat to: peer=3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 backoff time=10ms error="failed to connect to peer: 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6"
2025-06-26T05:14:31.757Z [WARN]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to contact: server-id=server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 time=51.8274ms
2025-06-26T05:14:31.757Z [WARN]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to contact quorum of nodes, stepping down
2025-06-26T05:14:31.757Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: entering follower state: follower="Node at 8dbecc6b-9568-19fa-3084-832b914ec3b0 [Follower]" leader-address= leader-id=
2025-06-26T05:14:31.757Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:31.765Z [ERROR] server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to appendEntries to: peer="{Voter server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}" error="failed to connect to peer: 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6"
2025-06-26T05:14:31.766Z [WARN]  server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: heartbeat timeout reached, starting election: last-leader-addr=8dbecc6b-9568-19fa-3084-832b914ec3b0 last-leader-id=server-8dbecc6b-9568-19fa-3084-832b914ec3b0
2025-06-26T05:14:31.766Z [INFO]  server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: entering candidate state: node="Node at 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 [Candidate]" term=3
2025-06-26T05:14:31.766Z [ERROR] server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: failed to make requestVote RPC: target="{Voter server-8dbecc6b-9568-19fa-3084-832b914ec3b0 8dbecc6b-9568-19fa-3084-832b914ec3b0}" error="failed to connect to peer: 8dbecc6b-9568-19fa-3084-832b914ec3b0" term=3
2025-06-26T05:14:31.829Z [WARN]  server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: Election timeout reached, restarting election
2025-06-26T05:14:31.829Z [INFO]  server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: entering candidate state: node="Node at 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 [Candidate]" term=4
2025-06-26T05:14:31.829Z [ERROR] server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: failed to make requestVote RPC: target="{Voter server-8dbecc6b-9568-19fa-3084-832b914ec3b0 8dbecc6b-9568-19fa-3084-832b914ec3b0}" error="failed to connect to peer: 8dbecc6b-9568-19fa-3084-832b914ec3b0" term=4
2025-06-26T05:14:31.830Z [WARN]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:31.830Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: entering candidate state: node="Node at 8dbecc6b-9568-19fa-3084-832b914ec3b0 [Candidate]" term=3
2025-06-26T05:14:31.830Z [ERROR] server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to make requestVote RPC: target="{Voter server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}" error="failed to connect to peer: 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6" term=3
2025-06-26T05:14:31.897Z [WARN]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: Election timeout reached, restarting election
2025-06-26T05:14:31.897Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: entering candidate state: node="Node at 8dbecc6b-9568-19fa-3084-832b914ec3b0 [Candidate]" term=4
2025-06-26T05:14:31.897Z [ERROR] server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to make requestVote RPC: target="{Voter server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}" error="failed to connect to peer: 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6" term=4
2025-06-26T05:14:31.925Z [WARN]  server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: Election timeout reached, restarting election
2025-06-26T05:14:31.925Z [INFO]  server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: entering candidate state: node="Node at 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 [Candidate]" term=5
2025-06-26T05:14:31.925Z [ERROR] server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: failed to make requestVote RPC: target="{Voter server-8dbecc6b-9568-19fa-3084-832b914ec3b0 8dbecc6b-9568-19fa-3084-832b914ec3b0}" error="failed to connect to peer: 8dbecc6b-9568-19fa-3084-832b914ec3b0" term=5
2025-06-26T05:14:31.938Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:31.830139403 +0000 UTC m=+99.979249681 (0 nodes), 72.37132ms from start of poll, 399.61764ms from cluster start. Timeout at 2025-06-26 05:14:31.938121243 +0000 UTC m=+100.087231521, 107.98184ms after stability
2025-06-26T05:14:31.977Z [WARN]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: Election timeout reached, restarting election
2025-06-26T05:14:31.977Z [INFO]  server-8dbecc6b-9568-19fa-3084-832b914ec3b0: entering candidate state: node="Node at 8dbecc6b-9568-19fa-3084-832b914ec3b0 [Candidate]" term=5
2025-06-26T05:14:31.977Z [ERROR] server-8dbecc6b-9568-19fa-3084-832b914ec3b0: failed to make requestVote RPC: target="{Voter server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6}" error="failed to connect to peer: 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6" term=5
2025-06-26T05:14:32.017Z [WARN]  server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: Election timeout reached, restarting election
2025-06-26T05:14:32.017Z [INFO]  server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: entering candidate state: node="Node at 3b2a69b9-3801-8cf8-a1b0-27fb7d441af6 [Candidate]" term=6
2025-06-26T05:14:32.017Z [ERROR] server-3b2a69b9-3801-8cf8-a1b0-27fb7d441af6: failed to make requestVote RPC: target="{Voter server-8dbecc6b-9568-19fa-3084-832b914ec3b0 8dbecc6b-9568-19fa-3084-832b914ec3b0}" error="failed to connect to peer: 8dbecc6b-9568-19fa-3084-832b914ec3b0" term=6
--- PASS: TestRaft_LeaderLeaseExpire (0.62s)
=== RUN   TestRaft_Barrier
2025-06-26T05:14:32.046Z [INFO]  server-1d98aa71-e505-064b-8bb7-63c6e99edb7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d98aa71-e505-064b-8bb7-63c6e99edb7a Address:1d98aa71-e505-064b-8bb7-63c6e99edb7a} {Suffrage:Voter ID:server-06a2bb11-441f-e1bf-1424-70efc059bd3a Address:06a2bb11-441f-e1bf-1424-70efc059bd3a} {Suffrage:Voter ID:server-9ce8136a-578f-e850-a353-4268d12929b6 Address:9ce8136a-578f-e850-a353-4268d12929b6}]"
2025-06-26T05:14:32.046Z [INFO]  server-06a2bb11-441f-e1bf-1424-70efc059bd3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d98aa71-e505-064b-8bb7-63c6e99edb7a Address:1d98aa71-e505-064b-8bb7-63c6e99edb7a} {Suffrage:Voter ID:server-06a2bb11-441f-e1bf-1424-70efc059bd3a Address:06a2bb11-441f-e1bf-1424-70efc059bd3a} {Suffrage:Voter ID:server-9ce8136a-578f-e850-a353-4268d12929b6 Address:9ce8136a-578f-e850-a353-4268d12929b6}]"
2025-06-26T05:14:32.046Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d98aa71-e505-064b-8bb7-63c6e99edb7a Address:1d98aa71-e505-064b-8bb7-63c6e99edb7a} {Suffrage:Voter ID:server-06a2bb11-441f-e1bf-1424-70efc059bd3a Address:06a2bb11-441f-e1bf-1424-70efc059bd3a} {Suffrage:Voter ID:server-9ce8136a-578f-e850-a353-4268d12929b6 Address:9ce8136a-578f-e850-a353-4268d12929b6}]"
2025-06-26T05:14:32.046Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:32.046Z [INFO]  server-1d98aa71-e505-064b-8bb7-63c6e99edb7a: entering follower state: follower="Node at 1d98aa71-e505-064b-8bb7-63c6e99edb7a [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.046Z [INFO]  server-06a2bb11-441f-e1bf-1424-70efc059bd3a: entering follower state: follower="Node at 06a2bb11-441f-e1bf-1424-70efc059bd3a [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.046Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: entering follower state: follower="Node at 9ce8136a-578f-e850-a353-4268d12929b6 [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.109Z [WARN]  server-9ce8136a-578f-e850-a353-4268d12929b6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:32.109Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: entering candidate state: node="Node at 9ce8136a-578f-e850-a353-4268d12929b6 [Candidate]" term=2
2025-06-26T05:14:32.109Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: election won: term=2 tally=2
2025-06-26T05:14:32.109Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: entering leader state: leader="Node at 9ce8136a-578f-e850-a353-4268d12929b6 [Leader]"
2025-06-26T05:14:32.109Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: added peer, starting replication: peer=server-1d98aa71-e505-064b-8bb7-63c6e99edb7a
2025-06-26T05:14:32.109Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: added peer, starting replication: peer=server-06a2bb11-441f-e1bf-1424-70efc059bd3a
2025-06-26T05:14:32.109Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: pipelining replication: peer="{Voter server-06a2bb11-441f-e1bf-1424-70efc059bd3a 06a2bb11-441f-e1bf-1424-70efc059bd3a}"
2025-06-26T05:14:32.110Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: pipelining replication: peer="{Voter server-1d98aa71-e505-064b-8bb7-63c6e99edb7a 1d98aa71-e505-064b-8bb7-63c6e99edb7a}"
2025-06-26T05:14:32.217Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:32.110291843 +0000 UTC m=+100.259402121 (1 nodes), 63.956ms from start of poll, 64.169ms from cluster start. Timeout at 2025-06-26 05:14:32.217600723 +0000 UTC m=+100.366711001, 107.30888ms after stability
2025-06-26T05:14:32.237Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: aborting pipeline replication: peer="{Voter server-06a2bb11-441f-e1bf-1424-70efc059bd3a 06a2bb11-441f-e1bf-1424-70efc059bd3a}"
2025-06-26T05:14:32.238Z [INFO]  server-9ce8136a-578f-e850-a353-4268d12929b6: aborting pipeline replication: peer="{Voter server-1d98aa71-e505-064b-8bb7-63c6e99edb7a 1d98aa71-e505-064b-8bb7-63c6e99edb7a}"
--- PASS: TestRaft_Barrier (0.19s)
=== RUN   TestRaft_VerifyLeader
2025-06-26T05:14:32.238Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5446775-7386-ea6e-4e0a-5176751a3b54 Address:a5446775-7386-ea6e-4e0a-5176751a3b54} {Suffrage:Voter ID:server-8f7e6121-098a-4b7f-1e96-7874faa683d7 Address:8f7e6121-098a-4b7f-1e96-7874faa683d7} {Suffrage:Voter ID:server-98513f0c-4de8-9a0c-3521-b4c7f4ca779f Address:98513f0c-4de8-9a0c-3521-b4c7f4ca779f}]"
2025-06-26T05:14:32.238Z [INFO]  server-8f7e6121-098a-4b7f-1e96-7874faa683d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5446775-7386-ea6e-4e0a-5176751a3b54 Address:a5446775-7386-ea6e-4e0a-5176751a3b54} {Suffrage:Voter ID:server-8f7e6121-098a-4b7f-1e96-7874faa683d7 Address:8f7e6121-098a-4b7f-1e96-7874faa683d7} {Suffrage:Voter ID:server-98513f0c-4de8-9a0c-3521-b4c7f4ca779f Address:98513f0c-4de8-9a0c-3521-b4c7f4ca779f}]"
2025-06-26T05:14:32.238Z [INFO]  server-98513f0c-4de8-9a0c-3521-b4c7f4ca779f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5446775-7386-ea6e-4e0a-5176751a3b54 Address:a5446775-7386-ea6e-4e0a-5176751a3b54} {Suffrage:Voter ID:server-8f7e6121-098a-4b7f-1e96-7874faa683d7 Address:8f7e6121-098a-4b7f-1e96-7874faa683d7} {Suffrage:Voter ID:server-98513f0c-4de8-9a0c-3521-b4c7f4ca779f Address:98513f0c-4de8-9a0c-3521-b4c7f4ca779f}]"
2025-06-26T05:14:32.238Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:32.238Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: entering follower state: follower="Node at a5446775-7386-ea6e-4e0a-5176751a3b54 [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.238Z [INFO]  server-8f7e6121-098a-4b7f-1e96-7874faa683d7: entering follower state: follower="Node at 8f7e6121-098a-4b7f-1e96-7874faa683d7 [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.238Z [INFO]  server-98513f0c-4de8-9a0c-3521-b4c7f4ca779f: entering follower state: follower="Node at 98513f0c-4de8-9a0c-3521-b4c7f4ca779f [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.293Z [WARN]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:32.293Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: entering candidate state: node="Node at a5446775-7386-ea6e-4e0a-5176751a3b54 [Candidate]" term=2
2025-06-26T05:14:32.293Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: election won: term=2 tally=2
2025-06-26T05:14:32.293Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: entering leader state: leader="Node at a5446775-7386-ea6e-4e0a-5176751a3b54 [Leader]"
2025-06-26T05:14:32.293Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: added peer, starting replication: peer=server-8f7e6121-098a-4b7f-1e96-7874faa683d7
2025-06-26T05:14:32.293Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: added peer, starting replication: peer=server-98513f0c-4de8-9a0c-3521-b4c7f4ca779f
2025-06-26T05:14:32.293Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: pipelining replication: peer="{Voter server-98513f0c-4de8-9a0c-3521-b4c7f4ca779f 98513f0c-4de8-9a0c-3521-b4c7f4ca779f}"
2025-06-26T05:14:32.293Z [WARN]  server-8f7e6121-098a-4b7f-1e96-7874faa683d7: rejecting vote request since our last term is greater: candidate=a5446775-7386-ea6e-4e0a-5176751a3b54 last-term=2 last-candidate-term=1
2025-06-26T05:14:32.293Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: pipelining replication: peer="{Voter server-8f7e6121-098a-4b7f-1e96-7874faa683d7 8f7e6121-098a-4b7f-1e96-7874faa683d7}"
2025-06-26T05:14:32.401Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:32.293947323 +0000 UTC m=+100.443057561 (1 nodes), 55.3252ms from start of poll, 55.55852ms from cluster start. Timeout at 2025-06-26 05:14:32.401604403 +0000 UTC m=+100.550714681, 107.65712ms after stability
2025-06-26T05:14:32.401Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: aborting pipeline replication: peer="{Voter server-98513f0c-4de8-9a0c-3521-b4c7f4ca779f 98513f0c-4de8-9a0c-3521-b4c7f4ca779f}"
2025-06-26T05:14:32.401Z [INFO]  server-a5446775-7386-ea6e-4e0a-5176751a3b54: aborting pipeline replication: peer="{Voter server-8f7e6121-098a-4b7f-1e96-7874faa683d7 8f7e6121-098a-4b7f-1e96-7874faa683d7}"
--- PASS: TestRaft_VerifyLeader (0.16s)
=== RUN   TestRaft_VerifyLeader_Single
2025-06-26T05:14:32.402Z [INFO]  server-cf58db4d-31e6-f9ea-07a5-4908a0dbd095: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf58db4d-31e6-f9ea-07a5-4908a0dbd095 Address:cf58db4d-31e6-f9ea-07a5-4908a0dbd095}]"
2025-06-26T05:14:32.402Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:32.402Z [INFO]  server-cf58db4d-31e6-f9ea-07a5-4908a0dbd095: entering follower state: follower="Node at cf58db4d-31e6-f9ea-07a5-4908a0dbd095 [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.485Z [WARN]  server-cf58db4d-31e6-f9ea-07a5-4908a0dbd095: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:32.485Z [INFO]  server-cf58db4d-31e6-f9ea-07a5-4908a0dbd095: entering candidate state: node="Node at cf58db4d-31e6-f9ea-07a5-4908a0dbd095 [Candidate]" term=2
2025-06-26T05:14:32.485Z [INFO]  server-cf58db4d-31e6-f9ea-07a5-4908a0dbd095: election won: term=2 tally=1
2025-06-26T05:14:32.485Z [INFO]  server-cf58db4d-31e6-f9ea-07a5-4908a0dbd095: entering leader state: leader="Node at cf58db4d-31e6-f9ea-07a5-4908a0dbd095 [Leader]"
2025-06-26T05:14:32.598Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:32.485764923 +0000 UTC m=+100.634875161 (1 nodes), 83.47784ms from start of poll, 83.58472ms from cluster start. Timeout at 2025-06-26 05:14:32.598508723 +0000 UTC m=+100.747619001, 112.74384ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.20s)
=== RUN   TestRaft_VerifyLeader_Fail
2025-06-26T05:14:32.598Z [INFO]  server-1d5e6a1a-f83e-8d3d-baa4-36732c46798f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d5e6a1a-f83e-8d3d-baa4-36732c46798f Address:1d5e6a1a-f83e-8d3d-baa4-36732c46798f} {Suffrage:Voter ID:server-68353714-7016-6449-694f-635d3d66696e Address:68353714-7016-6449-694f-635d3d66696e}]"
2025-06-26T05:14:32.599Z [INFO]  server-68353714-7016-6449-694f-635d3d66696e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d5e6a1a-f83e-8d3d-baa4-36732c46798f Address:1d5e6a1a-f83e-8d3d-baa4-36732c46798f} {Suffrage:Voter ID:server-68353714-7016-6449-694f-635d3d66696e Address:68353714-7016-6449-694f-635d3d66696e}]"
2025-06-26T05:14:32.599Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:32.599Z [INFO]  server-1d5e6a1a-f83e-8d3d-baa4-36732c46798f: entering follower state: follower="Node at 1d5e6a1a-f83e-8d3d-baa4-36732c46798f [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.599Z [INFO]  server-68353714-7016-6449-694f-635d3d66696e: entering follower state: follower="Node at 68353714-7016-6449-694f-635d3d66696e [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.662Z [WARN]  server-68353714-7016-6449-694f-635d3d66696e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:32.662Z [INFO]  server-68353714-7016-6449-694f-635d3d66696e: entering candidate state: node="Node at 68353714-7016-6449-694f-635d3d66696e [Candidate]" term=2
2025-06-26T05:14:32.662Z [INFO]  server-68353714-7016-6449-694f-635d3d66696e: election won: term=2 tally=2
2025-06-26T05:14:32.662Z [INFO]  server-68353714-7016-6449-694f-635d3d66696e: entering leader state: leader="Node at 68353714-7016-6449-694f-635d3d66696e [Leader]"
2025-06-26T05:14:32.662Z [INFO]  server-68353714-7016-6449-694f-635d3d66696e: added peer, starting replication: peer=server-1d5e6a1a-f83e-8d3d-baa4-36732c46798f
2025-06-26T05:14:32.662Z [INFO]  server-68353714-7016-6449-694f-635d3d66696e: pipelining replication: peer="{Voter server-1d5e6a1a-f83e-8d3d-baa4-36732c46798f 1d5e6a1a-f83e-8d3d-baa4-36732c46798f}"
2025-06-26T05:14:32.770Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:32.662927443 +0000 UTC m=+100.812037681 (1 nodes), 63.8562ms from start of poll, 64.02128ms from cluster start. Timeout at 2025-06-26 05:14:32.770015483 +0000 UTC m=+100.919125761, 107.08808ms after stability
2025-06-26T05:14:32.770Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:32.878Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:32.770076563 +0000 UTC m=+100.919186841 (1 nodes), 640ns from start of poll, 171.17044ms from cluster start. Timeout at 2025-06-26 05:14:32.877957163 +0000 UTC m=+101.027067441, 107.8806ms after stability
2025-06-26T05:14:32.881Z [ERROR] server-68353714-7016-6449-694f-635d3d66696e: peer has newer term, stopping replication: peer="{Voter server-1d5e6a1a-f83e-8d3d-baa4-36732c46798f 1d5e6a1a-f83e-8d3d-baa4-36732c46798f}"
2025-06-26T05:14:32.881Z [INFO]  server-68353714-7016-6449-694f-635d3d66696e: aborting pipeline replication: peer="{Voter server-1d5e6a1a-f83e-8d3d-baa4-36732c46798f 1d5e6a1a-f83e-8d3d-baa4-36732c46798f}"
2025-06-26T05:14:32.881Z [INFO]  server-68353714-7016-6449-694f-635d3d66696e: entering follower state: follower="Node at 68353714-7016-6449-694f-635d3d66696e [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.28s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2025-06-26T05:14:32.882Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf14b4ac-f674-5131-07ae-a152818b247f Address:cf14b4ac-f674-5131-07ae-a152818b247f} {Suffrage:Voter ID:server-dab074c7-fddb-1e38-7c75-e4ee21035246 Address:dab074c7-fddb-1e38-7c75-e4ee21035246} {Suffrage:Voter ID:server-fd9ce02b-ab73-1031-c9c2-f451ac42ce62 Address:fd9ce02b-ab73-1031-c9c2-f451ac42ce62}]"
2025-06-26T05:14:32.882Z [INFO]  server-dab074c7-fddb-1e38-7c75-e4ee21035246: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf14b4ac-f674-5131-07ae-a152818b247f Address:cf14b4ac-f674-5131-07ae-a152818b247f} {Suffrage:Voter ID:server-dab074c7-fddb-1e38-7c75-e4ee21035246 Address:dab074c7-fddb-1e38-7c75-e4ee21035246} {Suffrage:Voter ID:server-fd9ce02b-ab73-1031-c9c2-f451ac42ce62 Address:fd9ce02b-ab73-1031-c9c2-f451ac42ce62}]"
2025-06-26T05:14:32.882Z [INFO]  server-fd9ce02b-ab73-1031-c9c2-f451ac42ce62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf14b4ac-f674-5131-07ae-a152818b247f Address:cf14b4ac-f674-5131-07ae-a152818b247f} {Suffrage:Voter ID:server-dab074c7-fddb-1e38-7c75-e4ee21035246 Address:dab074c7-fddb-1e38-7c75-e4ee21035246} {Suffrage:Voter ID:server-fd9ce02b-ab73-1031-c9c2-f451ac42ce62 Address:fd9ce02b-ab73-1031-c9c2-f451ac42ce62}]"
2025-06-26T05:14:32.882Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:32.882Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: entering follower state: follower="Node at cf14b4ac-f674-5131-07ae-a152818b247f [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.882Z [INFO]  server-dab074c7-fddb-1e38-7c75-e4ee21035246: entering follower state: follower="Node at dab074c7-fddb-1e38-7c75-e4ee21035246 [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.882Z [INFO]  server-fd9ce02b-ab73-1031-c9c2-f451ac42ce62: entering follower state: follower="Node at fd9ce02b-ab73-1031-c9c2-f451ac42ce62 [Follower]" leader-address= leader-id=
2025-06-26T05:14:32.937Z [WARN]  server-cf14b4ac-f674-5131-07ae-a152818b247f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:32.937Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: entering candidate state: node="Node at cf14b4ac-f674-5131-07ae-a152818b247f [Candidate]" term=2
2025-06-26T05:14:32.938Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: election won: term=2 tally=2
2025-06-26T05:14:32.938Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: entering leader state: leader="Node at cf14b4ac-f674-5131-07ae-a152818b247f [Leader]"
2025-06-26T05:14:32.938Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: added peer, starting replication: peer=server-dab074c7-fddb-1e38-7c75-e4ee21035246
2025-06-26T05:14:32.938Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: added peer, starting replication: peer=server-fd9ce02b-ab73-1031-c9c2-f451ac42ce62
2025-06-26T05:14:32.938Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: pipelining replication: peer="{Voter server-fd9ce02b-ab73-1031-c9c2-f451ac42ce62 fd9ce02b-ab73-1031-c9c2-f451ac42ce62}"
2025-06-26T05:14:32.938Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: pipelining replication: peer="{Voter server-dab074c7-fddb-1e38-7c75-e4ee21035246 dab074c7-fddb-1e38-7c75-e4ee21035246}"
2025-06-26T05:14:33.045Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:32.938089763 +0000 UTC m=+101.087200041 (1 nodes), 55.51136ms from start of poll, 55.73608ms from cluster start. Timeout at 2025-06-26 05:14:33.045605683 +0000 UTC m=+101.194715961, 107.51592ms after stability
2025-06-26T05:14:33.053Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:33.161Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:33.053645123 +0000 UTC m=+101.202755401 (2 nodes), 840ns from start of poll, 171.29144ms from cluster start. Timeout at 2025-06-26 05:14:33.161902203 +0000 UTC m=+101.311012481, 108.25708ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at dab074c7-fddb-1e38-7c75-e4ee21035246 [Follower]
2025-06-26T05:14:33.162Z [ERROR] server-cf14b4ac-f674-5131-07ae-a152818b247f: failed to heartbeat to: peer=dab074c7-fddb-1e38-7c75-e4ee21035246 backoff time=10ms error="failed to connect to peer: dab074c7-fddb-1e38-7c75-e4ee21035246"
2025-06-26T05:14:33.162Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: aborting pipeline replication: peer="{Voter server-fd9ce02b-ab73-1031-c9c2-f451ac42ce62 fd9ce02b-ab73-1031-c9c2-f451ac42ce62}"
2025-06-26T05:14:33.162Z [INFO]  server-cf14b4ac-f674-5131-07ae-a152818b247f: aborting pipeline replication: peer="{Voter server-dab074c7-fddb-1e38-7c75-e4ee21035246 dab074c7-fddb-1e38-7c75-e4ee21035246}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s)
=== RUN   TestRaft_NotifyCh
2025-06-26T05:14:33.162Z [INFO]  server-aefec618-2129-5451-b814-2b8a90b08db3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aefec618-2129-5451-b814-2b8a90b08db3 Address:aefec618-2129-5451-b814-2b8a90b08db3}]"
2025-06-26T05:14:33.162Z [INFO]  server-aefec618-2129-5451-b814-2b8a90b08db3: entering follower state: follower="Node at aefec618-2129-5451-b814-2b8a90b08db3 [Follower]" leader-address= leader-id=
2025-06-26T05:14:33.213Z [WARN]  server-aefec618-2129-5451-b814-2b8a90b08db3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:33.213Z [INFO]  server-aefec618-2129-5451-b814-2b8a90b08db3: entering candidate state: node="Node at aefec618-2129-5451-b814-2b8a90b08db3 [Candidate]" term=2
2025-06-26T05:14:33.213Z [INFO]  server-aefec618-2129-5451-b814-2b8a90b08db3: election won: term=2 tally=1
2025-06-26T05:14:33.213Z [INFO]  server-aefec618-2129-5451-b814-2b8a90b08db3: entering leader state: leader="Node at aefec618-2129-5451-b814-2b8a90b08db3 [Leader]"
--- PASS: TestRaft_NotifyCh (0.05s)
=== RUN   TestRaft_AppendEntry
2025-06-26T05:14:33.214Z [INFO]  server-9a365c2a-f4c4-b4c7-965b-23510a062986: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a365c2a-f4c4-b4c7-965b-23510a062986 Address:9a365c2a-f4c4-b4c7-965b-23510a062986} {Suffrage:Voter ID:server-5b5551b2-f880-e261-a266-0270f880cb29 Address:5b5551b2-f880-e261-a266-0270f880cb29} {Suffrage:Voter ID:server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b Address:47d7a1e2-e35b-ffb3-3808-e29f5e29f29b}]"
2025-06-26T05:14:33.214Z [INFO]  server-5b5551b2-f880-e261-a266-0270f880cb29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a365c2a-f4c4-b4c7-965b-23510a062986 Address:9a365c2a-f4c4-b4c7-965b-23510a062986} {Suffrage:Voter ID:server-5b5551b2-f880-e261-a266-0270f880cb29 Address:5b5551b2-f880-e261-a266-0270f880cb29} {Suffrage:Voter ID:server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b Address:47d7a1e2-e35b-ffb3-3808-e29f5e29f29b}]"
2025-06-26T05:14:33.214Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a365c2a-f4c4-b4c7-965b-23510a062986 Address:9a365c2a-f4c4-b4c7-965b-23510a062986} {Suffrage:Voter ID:server-5b5551b2-f880-e261-a266-0270f880cb29 Address:5b5551b2-f880-e261-a266-0270f880cb29} {Suffrage:Voter ID:server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b Address:47d7a1e2-e35b-ffb3-3808-e29f5e29f29b}]"
2025-06-26T05:14:33.214Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:33.214Z [INFO]  server-9a365c2a-f4c4-b4c7-965b-23510a062986: entering follower state: follower="Node at 9a365c2a-f4c4-b4c7-965b-23510a062986 [Follower]" leader-address= leader-id=
2025-06-26T05:14:33.214Z [INFO]  server-5b5551b2-f880-e261-a266-0270f880cb29: entering follower state: follower="Node at 5b5551b2-f880-e261-a266-0270f880cb29 [Follower]" leader-address= leader-id=
2025-06-26T05:14:33.214Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: entering follower state: follower="Node at 47d7a1e2-e35b-ffb3-3808-e29f5e29f29b [Follower]" leader-address= leader-id=
2025-06-26T05:14:33.289Z [WARN]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:33.289Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: entering candidate state: node="Node at 47d7a1e2-e35b-ffb3-3808-e29f5e29f29b [Candidate]" term=2
2025-06-26T05:14:33.289Z [WARN]  server-5b5551b2-f880-e261-a266-0270f880cb29: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:33.289Z [INFO]  server-5b5551b2-f880-e261-a266-0270f880cb29: entering candidate state: node="Node at 5b5551b2-f880-e261-a266-0270f880cb29 [Candidate]" term=2
2025-06-26T05:14:33.289Z [INFO]  server-5b5551b2-f880-e261-a266-0270f880cb29: duplicate requestVote for same term: term=2
2025-06-26T05:14:33.289Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: election won: term=2 tally=2
2025-06-26T05:14:33.289Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: entering leader state: leader="Node at 47d7a1e2-e35b-ffb3-3808-e29f5e29f29b [Leader]"
2025-06-26T05:14:33.289Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: added peer, starting replication: peer=server-9a365c2a-f4c4-b4c7-965b-23510a062986
2025-06-26T05:14:33.289Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: added peer, starting replication: peer=server-5b5551b2-f880-e261-a266-0270f880cb29
2025-06-26T05:14:33.290Z [INFO]  server-5b5551b2-f880-e261-a266-0270f880cb29: entering follower state: follower="Node at 5b5551b2-f880-e261-a266-0270f880cb29 [Follower]" leader-address=47d7a1e2-e35b-ffb3-3808-e29f5e29f29b leader-id=server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b
2025-06-26T05:14:33.290Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: pipelining replication: peer="{Voter server-5b5551b2-f880-e261-a266-0270f880cb29 5b5551b2-f880-e261-a266-0270f880cb29}"
2025-06-26T05:14:33.290Z [INFO]  server-9a365c2a-f4c4-b4c7-965b-23510a062986: duplicate requestVote for same term: term=2
2025-06-26T05:14:33.290Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: pipelining replication: peer="{Voter server-9a365c2a-f4c4-b4c7-965b-23510a062986 9a365c2a-f4c4-b4c7-965b-23510a062986}"
2025-06-26T05:14:33.290Z [WARN]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: rejecting vote request since we have a leader: from=5b5551b2-f880-e261-a266-0270f880cb29 leader=47d7a1e2-e35b-ffb3-3808-e29f5e29f29b leader-id=server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b
2025-06-26T05:14:33.398Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:33.290330323 +0000 UTC m=+101.439440601 (2 nodes), 75.95116ms from start of poll, 76.16108ms from cluster start. Timeout at 2025-06-26 05:14:33.397869003 +0000 UTC m=+101.546979281, 107.53868ms after stability
2025-06-26T05:14:33.398Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:33.505Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:33.398161403 +0000 UTC m=+101.547271681 (1 nodes), 1.72µs from start of poll, 183.99216ms from cluster start. Timeout at 2025-06-26 05:14:33.505607963 +0000 UTC m=+101.654718201, 107.44652ms after stability
2025-06-26T05:14:33.505Z [WARN]  server-9a365c2a-f4c4-b4c7-965b-23510a062986: clearing log suffix: from=1 to=2
2025-06-26T05:14:33.505Z [WARN]  server-9a365c2a-f4c4-b4c7-965b-23510a062986: skipping application of old log: index=1
2025-06-26T05:14:33.505Z [WARN]  server-9a365c2a-f4c4-b4c7-965b-23510a062986: skipping application of old log: index=1
2025-06-26T05:14:33.505Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: aborting pipeline replication: peer="{Voter server-5b5551b2-f880-e261-a266-0270f880cb29 5b5551b2-f880-e261-a266-0270f880cb29}"
2025-06-26T05:14:33.505Z [INFO]  server-47d7a1e2-e35b-ffb3-3808-e29f5e29f29b: aborting pipeline replication: peer="{Voter server-9a365c2a-f4c4-b4c7-965b-23510a062986 9a365c2a-f4c4-b4c7-965b-23510a062986}"
--- PASS: TestRaft_AppendEntry (0.29s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2025-06-26T05:14:33.506Z [INFO]  server-1b7685b7-16a4-7544-c6cf-973e53d6e3e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b7685b7-16a4-7544-c6cf-973e53d6e3e8 Address:1b7685b7-16a4-7544-c6cf-973e53d6e3e8} {Suffrage:Voter ID:server-722a1e29-a820-5fc2-5799-3543ea0d9521 Address:722a1e29-a820-5fc2-5799-3543ea0d9521} {Suffrage:Voter ID:server-46b975b9-5de0-253b-af30-129107d79f5c Address:46b975b9-5de0-253b-af30-129107d79f5c}]"
2025-06-26T05:14:33.506Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b7685b7-16a4-7544-c6cf-973e53d6e3e8 Address:1b7685b7-16a4-7544-c6cf-973e53d6e3e8} {Suffrage:Voter ID:server-722a1e29-a820-5fc2-5799-3543ea0d9521 Address:722a1e29-a820-5fc2-5799-3543ea0d9521} {Suffrage:Voter ID:server-46b975b9-5de0-253b-af30-129107d79f5c Address:46b975b9-5de0-253b-af30-129107d79f5c}]"
2025-06-26T05:14:33.506Z [INFO]  server-46b975b9-5de0-253b-af30-129107d79f5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b7685b7-16a4-7544-c6cf-973e53d6e3e8 Address:1b7685b7-16a4-7544-c6cf-973e53d6e3e8} {Suffrage:Voter ID:server-722a1e29-a820-5fc2-5799-3543ea0d9521 Address:722a1e29-a820-5fc2-5799-3543ea0d9521} {Suffrage:Voter ID:server-46b975b9-5de0-253b-af30-129107d79f5c Address:46b975b9-5de0-253b-af30-129107d79f5c}]"
2025-06-26T05:14:33.506Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:33.506Z [INFO]  server-1b7685b7-16a4-7544-c6cf-973e53d6e3e8: entering follower state: follower="Node at 1b7685b7-16a4-7544-c6cf-973e53d6e3e8 [Follower]" leader-address= leader-id=
2025-06-26T05:14:33.506Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: entering follower state: follower="Node at 722a1e29-a820-5fc2-5799-3543ea0d9521 [Follower]" leader-address= leader-id=
2025-06-26T05:14:33.506Z [INFO]  server-46b975b9-5de0-253b-af30-129107d79f5c: entering follower state: follower="Node at 46b975b9-5de0-253b-af30-129107d79f5c [Follower]" leader-address= leader-id=
2025-06-26T05:14:33.594Z [WARN]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:33.594Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: entering candidate state: node="Node at 722a1e29-a820-5fc2-5799-3543ea0d9521 [Candidate]" term=2
2025-06-26T05:14:33.594Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: election won: term=2 tally=2
2025-06-26T05:14:33.594Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: entering leader state: leader="Node at 722a1e29-a820-5fc2-5799-3543ea0d9521 [Leader]"
2025-06-26T05:14:33.594Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: added peer, starting replication: peer=server-1b7685b7-16a4-7544-c6cf-973e53d6e3e8
2025-06-26T05:14:33.594Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: added peer, starting replication: peer=server-46b975b9-5de0-253b-af30-129107d79f5c
2025-06-26T05:14:33.594Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: pipelining replication: peer="{Voter server-46b975b9-5de0-253b-af30-129107d79f5c 46b975b9-5de0-253b-af30-129107d79f5c}"
2025-06-26T05:14:33.594Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: pipelining replication: peer="{Voter server-1b7685b7-16a4-7544-c6cf-973e53d6e3e8 1b7685b7-16a4-7544-c6cf-973e53d6e3e8}"
2025-06-26T05:14:33.701Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:33.594323243 +0000 UTC m=+101.743433481 (2 nodes), 87.85576ms from start of poll, 88.06204ms from cluster start. Timeout at 2025-06-26 05:14:33.701585403 +0000 UTC m=+101.850695641, 107.26216ms after stability
2025-06-26T05:14:33.701Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:33.809Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:33.701657323 +0000 UTC m=+101.850767601 (1 nodes), 1.08µs from start of poll, 195.39616ms from cluster start. Timeout at 2025-06-26 05:14:33.809727763 +0000 UTC m=+101.958838081, 108.07048ms after stability
2025-06-26T05:14:33.809Z [WARN]  server-46b975b9-5de0-253b-af30-129107d79f5c: rejecting vote request since we have a leader: from=1b7685b7-16a4-7544-c6cf-973e53d6e3e8 leader=722a1e29-a820-5fc2-5799-3543ea0d9521 leader-id=server-722a1e29-a820-5fc2-5799-3543ea0d9521
2025-06-26T05:14:33.809Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: aborting pipeline replication: peer="{Voter server-46b975b9-5de0-253b-af30-129107d79f5c 46b975b9-5de0-253b-af30-129107d79f5c}"
2025-06-26T05:14:33.809Z [INFO]  server-722a1e29-a820-5fc2-5799-3543ea0d9521: aborting pipeline replication: peer="{Voter server-1b7685b7-16a4-7544-c6cf-973e53d6e3e8 1b7685b7-16a4-7544-c6cf-973e53d6e3e8}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.30s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2025-06-26T05:14:33.810Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9 Address:96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9} {Suffrage:Voter ID:server-d82be72c-3fc2-6834-1a0d-cb71fc3897e0 Address:d82be72c-3fc2-6834-1a0d-cb71fc3897e0} {Suffrage:Voter ID:server-88675b3d-f3fe-f4ef-b416-9c8566f1a843 Address:88675b3d-f3fe-f4ef-b416-9c8566f1a843}]"
2025-06-26T05:14:33.810Z [INFO]  server-d82be72c-3fc2-6834-1a0d-cb71fc3897e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9 Address:96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9} {Suffrage:Voter ID:server-d82be72c-3fc2-6834-1a0d-cb71fc3897e0 Address:d82be72c-3fc2-6834-1a0d-cb71fc3897e0} {Suffrage:Voter ID:server-88675b3d-f3fe-f4ef-b416-9c8566f1a843 Address:88675b3d-f3fe-f4ef-b416-9c8566f1a843}]"
2025-06-26T05:14:33.810Z [INFO]  server-88675b3d-f3fe-f4ef-b416-9c8566f1a843: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9 Address:96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9} {Suffrage:Voter ID:server-d82be72c-3fc2-6834-1a0d-cb71fc3897e0 Address:d82be72c-3fc2-6834-1a0d-cb71fc3897e0} {Suffrage:Voter ID:server-88675b3d-f3fe-f4ef-b416-9c8566f1a843 Address:88675b3d-f3fe-f4ef-b416-9c8566f1a843}]"
2025-06-26T05:14:33.810Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:33.810Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: entering follower state: follower="Node at 96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9 [Follower]" leader-address= leader-id=
2025-06-26T05:14:33.810Z [INFO]  server-d82be72c-3fc2-6834-1a0d-cb71fc3897e0: entering follower state: follower="Node at d82be72c-3fc2-6834-1a0d-cb71fc3897e0 [Follower]" leader-address= leader-id=
2025-06-26T05:14:33.810Z [INFO]  server-88675b3d-f3fe-f4ef-b416-9c8566f1a843: entering follower state: follower="Node at 88675b3d-f3fe-f4ef-b416-9c8566f1a843 [Follower]" leader-address= leader-id=
2025-06-26T05:14:33.865Z [WARN]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:33.865Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: entering candidate state: node="Node at 96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9 [Candidate]" term=2
2025-06-26T05:14:33.865Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: election won: term=2 tally=2
2025-06-26T05:14:33.865Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: entering leader state: leader="Node at 96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9 [Leader]"
2025-06-26T05:14:33.865Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: added peer, starting replication: peer=server-d82be72c-3fc2-6834-1a0d-cb71fc3897e0
2025-06-26T05:14:33.865Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: added peer, starting replication: peer=server-88675b3d-f3fe-f4ef-b416-9c8566f1a843
2025-06-26T05:14:33.865Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: pipelining replication: peer="{Voter server-88675b3d-f3fe-f4ef-b416-9c8566f1a843 88675b3d-f3fe-f4ef-b416-9c8566f1a843}"
2025-06-26T05:14:33.866Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: pipelining replication: peer="{Voter server-d82be72c-3fc2-6834-1a0d-cb71fc3897e0 d82be72c-3fc2-6834-1a0d-cb71fc3897e0}"
2025-06-26T05:14:33.977Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:33.866077443 +0000 UTC m=+102.015187721 (2 nodes), 55.5508ms from start of poll, 55.78136ms from cluster start. Timeout at 2025-06-26 05:14:33.977602123 +0000 UTC m=+102.126712401, 111.52468ms after stability
2025-06-26T05:14:33.977Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:34.085Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:33.977718243 +0000 UTC m=+102.126828521 (1 nodes), 1.8µs from start of poll, 167.42216ms from cluster start. Timeout at 2025-06-26 05:14:34.085574683 +0000 UTC m=+102.234684961, 107.85644ms after stability
2025-06-26T05:14:34.085Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: aborting pipeline replication: peer="{Voter server-88675b3d-f3fe-f4ef-b416-9c8566f1a843 88675b3d-f3fe-f4ef-b416-9c8566f1a843}"
2025-06-26T05:14:34.085Z [INFO]  server-96f4fe78-9856-c15f-1ed2-ede2a7f5dcd9: aborting pipeline replication: peer="{Voter server-d82be72c-3fc2-6834-1a0d-cb71fc3897e0 d82be72c-3fc2-6834-1a0d-cb71fc3897e0}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.28s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2025-06-26T05:14:34.086Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e50da6d7-f0e2-a90b-9374-b230cd127f5d Address:e50da6d7-f0e2-a90b-9374-b230cd127f5d} {Suffrage:Voter ID:server-228c638c-5357-6579-6bea-d3e03c4efef5 Address:228c638c-5357-6579-6bea-d3e03c4efef5} {Suffrage:Voter ID:server-375b3baa-dca7-8c8a-e9f8-420875aec2ce Address:375b3baa-dca7-8c8a-e9f8-420875aec2ce}]"
2025-06-26T05:14:34.086Z [INFO]  server-228c638c-5357-6579-6bea-d3e03c4efef5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e50da6d7-f0e2-a90b-9374-b230cd127f5d Address:e50da6d7-f0e2-a90b-9374-b230cd127f5d} {Suffrage:Voter ID:server-228c638c-5357-6579-6bea-d3e03c4efef5 Address:228c638c-5357-6579-6bea-d3e03c4efef5} {Suffrage:Voter ID:server-375b3baa-dca7-8c8a-e9f8-420875aec2ce Address:375b3baa-dca7-8c8a-e9f8-420875aec2ce}]"
2025-06-26T05:14:34.086Z [INFO]  server-375b3baa-dca7-8c8a-e9f8-420875aec2ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e50da6d7-f0e2-a90b-9374-b230cd127f5d Address:e50da6d7-f0e2-a90b-9374-b230cd127f5d} {Suffrage:Voter ID:server-228c638c-5357-6579-6bea-d3e03c4efef5 Address:228c638c-5357-6579-6bea-d3e03c4efef5} {Suffrage:Voter ID:server-375b3baa-dca7-8c8a-e9f8-420875aec2ce Address:375b3baa-dca7-8c8a-e9f8-420875aec2ce}]"
2025-06-26T05:14:34.086Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:34.086Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: entering follower state: follower="Node at e50da6d7-f0e2-a90b-9374-b230cd127f5d [Follower]" leader-address= leader-id=
2025-06-26T05:14:34.086Z [INFO]  server-228c638c-5357-6579-6bea-d3e03c4efef5: entering follower state: follower="Node at 228c638c-5357-6579-6bea-d3e03c4efef5 [Follower]" leader-address= leader-id=
2025-06-26T05:14:34.086Z [INFO]  server-375b3baa-dca7-8c8a-e9f8-420875aec2ce: entering follower state: follower="Node at 375b3baa-dca7-8c8a-e9f8-420875aec2ce [Follower]" leader-address= leader-id=
2025-06-26T05:14:34.153Z [WARN]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:34.153Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: entering candidate state: node="Node at e50da6d7-f0e2-a90b-9374-b230cd127f5d [Candidate]" term=2
2025-06-26T05:14:34.153Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: election won: term=2 tally=2
2025-06-26T05:14:34.153Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: entering leader state: leader="Node at e50da6d7-f0e2-a90b-9374-b230cd127f5d [Leader]"
2025-06-26T05:14:34.153Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: added peer, starting replication: peer=server-228c638c-5357-6579-6bea-d3e03c4efef5
2025-06-26T05:14:34.153Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: added peer, starting replication: peer=server-375b3baa-dca7-8c8a-e9f8-420875aec2ce
2025-06-26T05:14:34.153Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: pipelining replication: peer="{Voter server-375b3baa-dca7-8c8a-e9f8-420875aec2ce 375b3baa-dca7-8c8a-e9f8-420875aec2ce}"
2025-06-26T05:14:34.153Z [WARN]  server-375b3baa-dca7-8c8a-e9f8-420875aec2ce: rejecting vote request since our last term is greater: candidate=e50da6d7-f0e2-a90b-9374-b230cd127f5d last-term=2 last-candidate-term=1
2025-06-26T05:14:34.153Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: pipelining replication: peer="{Voter server-228c638c-5357-6579-6bea-d3e03c4efef5 228c638c-5357-6579-6bea-d3e03c4efef5}"
2025-06-26T05:14:34.262Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:34.153917763 +0000 UTC m=+102.303028041 (1 nodes), 67.54436ms from start of poll, 67.74456ms from cluster start. Timeout at 2025-06-26 05:14:34.262042883 +0000 UTC m=+102.411153161, 108.12512ms after stability
2025-06-26T05:14:34.262Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:34.371Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:34.262151483 +0000 UTC m=+102.411261761 (1 nodes), 520ns from start of poll, 175.97828ms from cluster start. Timeout at 2025-06-26 05:14:34.370976203 +0000 UTC m=+102.520086441, 108.82468ms after stability
2025-06-26T05:14:34.371Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:34.477Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:34.371178003 +0000 UTC m=+102.520288281 (1 nodes), 960ns from start of poll, 285.0048ms from cluster start. Timeout at 2025-06-26 05:14:34.477571723 +0000 UTC m=+102.626682001, 106.39372ms after stability
2025-06-26T05:14:34.477Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:34.585Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:34.477731443 +0000 UTC m=+102.626841721 (1 nodes), 520ns from start of poll, 391.55824ms from cluster start. Timeout at 2025-06-26 05:14:34.585649483 +0000 UTC m=+102.734759761, 107.91804ms after stability
2025-06-26T05:14:34.585Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:34.697Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:34.585878163 +0000 UTC m=+102.734988441 (1 nodes), 720ns from start of poll, 499.70496ms from cluster start. Timeout at 2025-06-26 05:14:34.697593283 +0000 UTC m=+102.846703561, 111.71512ms after stability
2025-06-26T05:14:34.697Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:34.805Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:34.697764643 +0000 UTC m=+102.846874921 (1 nodes), 560ns from start of poll, 611.59144ms from cluster start. Timeout at 2025-06-26 05:14:34.805615763 +0000 UTC m=+102.954726001, 107.85108ms after stability
2025-06-26T05:14:34.913Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:35.026Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:34.913749883 +0000 UTC m=+103.062860161 (1 nodes), 1.04µs from start of poll, 827.57668ms from cluster start. Timeout at 2025-06-26 05:14:35.025979283 +0000 UTC m=+103.175089641, 112.22948ms after stability
2025-06-26T05:14:35.026Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: aborting pipeline replication: peer="{Voter server-375b3baa-dca7-8c8a-e9f8-420875aec2ce 375b3baa-dca7-8c8a-e9f8-420875aec2ce}"
2025-06-26T05:14:35.026Z [INFO]  server-e50da6d7-f0e2-a90b-9374-b230cd127f5d: aborting pipeline replication: peer="{Voter server-228c638c-5357-6579-6bea-d3e03c4efef5 228c638c-5357-6579-6bea-d3e03c4efef5}"
--- PASS: TestRaft_LeaderID_Propagated (0.94s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2025-06-26T05:14:35.026Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f1b7583-0405-5c4b-afd9-96a6bc85355d Address:8f1b7583-0405-5c4b-afd9-96a6bc85355d} {Suffrage:Voter ID:server-7419a4e7-0067-7934-5680-dce6362e028d Address:7419a4e7-0067-7934-5680-dce6362e028d} {Suffrage:Voter ID:server-88fe5755-daa0-db68-2267-849ed9724abf Address:88fe5755-daa0-db68-2267-849ed9724abf}]"
2025-06-26T05:14:35.026Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f1b7583-0405-5c4b-afd9-96a6bc85355d Address:8f1b7583-0405-5c4b-afd9-96a6bc85355d} {Suffrage:Voter ID:server-7419a4e7-0067-7934-5680-dce6362e028d Address:7419a4e7-0067-7934-5680-dce6362e028d} {Suffrage:Voter ID:server-88fe5755-daa0-db68-2267-849ed9724abf Address:88fe5755-daa0-db68-2267-849ed9724abf}]"
2025-06-26T05:14:35.026Z [INFO]  server-88fe5755-daa0-db68-2267-849ed9724abf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f1b7583-0405-5c4b-afd9-96a6bc85355d Address:8f1b7583-0405-5c4b-afd9-96a6bc85355d} {Suffrage:Voter ID:server-7419a4e7-0067-7934-5680-dce6362e028d Address:7419a4e7-0067-7934-5680-dce6362e028d} {Suffrage:Voter ID:server-88fe5755-daa0-db68-2267-849ed9724abf Address:88fe5755-daa0-db68-2267-849ed9724abf}]"
2025-06-26T05:14:35.026Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:35.026Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: entering follower state: follower="Node at 8f1b7583-0405-5c4b-afd9-96a6bc85355d [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.026Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: entering follower state: follower="Node at 7419a4e7-0067-7934-5680-dce6362e028d [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.026Z [INFO]  server-88fe5755-daa0-db68-2267-849ed9724abf: entering follower state: follower="Node at 88fe5755-daa0-db68-2267-849ed9724abf [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.081Z [WARN]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:35.081Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: entering candidate state: node="Node at 8f1b7583-0405-5c4b-afd9-96a6bc85355d [Candidate]" term=2
2025-06-26T05:14:35.081Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: election won: term=2 tally=2
2025-06-26T05:14:35.081Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: entering leader state: leader="Node at 8f1b7583-0405-5c4b-afd9-96a6bc85355d [Leader]"
2025-06-26T05:14:35.081Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: added peer, starting replication: peer=server-7419a4e7-0067-7934-5680-dce6362e028d
2025-06-26T05:14:35.081Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: added peer, starting replication: peer=server-88fe5755-daa0-db68-2267-849ed9724abf
2025-06-26T05:14:35.081Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: pipelining replication: peer="{Voter server-88fe5755-daa0-db68-2267-849ed9724abf 88fe5755-daa0-db68-2267-849ed9724abf}"
2025-06-26T05:14:35.081Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: pipelining replication: peer="{Voter server-7419a4e7-0067-7934-5680-dce6362e028d 7419a4e7-0067-7934-5680-dce6362e028d}"
2025-06-26T05:14:35.189Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:35.082095323 +0000 UTC m=+103.231205601 (1 nodes), 55.33436ms from start of poll, 55.57896ms from cluster start. Timeout at 2025-06-26 05:14:35.189620963 +0000 UTC m=+103.338731201, 107.5256ms after stability
2025-06-26T05:14:35.189Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:35.298Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:35.189697643 +0000 UTC m=+103.338807921 (1 nodes), 920ns from start of poll, 163.18128ms from cluster start. Timeout at 2025-06-26 05:14:35.297968883 +0000 UTC m=+103.447079161, 108.27124ms after stability
2025-06-26T05:14:35.298Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: entering candidate state: node="Node at 7419a4e7-0067-7934-5680-dce6362e028d [Candidate]" term=3
2025-06-26T05:14:35.298Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: election won: term=3 tally=2
2025-06-26T05:14:35.298Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: entering leader state: leader="Node at 7419a4e7-0067-7934-5680-dce6362e028d [Leader]"
2025-06-26T05:14:35.298Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: added peer, starting replication: peer=server-8f1b7583-0405-5c4b-afd9-96a6bc85355d
2025-06-26T05:14:35.298Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: added peer, starting replication: peer=server-88fe5755-daa0-db68-2267-849ed9724abf
2025-06-26T05:14:35.298Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:35.298Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: entering follower state: follower="Node at 8f1b7583-0405-5c4b-afd9-96a6bc85355d [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.298Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: aborting pipeline replication: peer="{Voter server-88fe5755-daa0-db68-2267-849ed9724abf 88fe5755-daa0-db68-2267-849ed9724abf}"
2025-06-26T05:14:35.298Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: pipelining replication: peer="{Voter server-8f1b7583-0405-5c4b-afd9-96a6bc85355d 8f1b7583-0405-5c4b-afd9-96a6bc85355d}"
2025-06-26T05:14:35.298Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: pipelining replication: peer="{Voter server-88fe5755-daa0-db68-2267-849ed9724abf 88fe5755-daa0-db68-2267-849ed9724abf}"
2025-06-26T05:14:35.298Z [INFO]  server-8f1b7583-0405-5c4b-afd9-96a6bc85355d: aborting pipeline replication: peer="{Voter server-7419a4e7-0067-7934-5680-dce6362e028d 7419a4e7-0067-7934-5680-dce6362e028d}"
2025-06-26T05:14:35.405Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:35.298445763 +0000 UTC m=+103.447556041 (1 nodes), 180.64µs from start of poll, 271.9294ms from cluster start. Timeout at 2025-06-26 05:14:35.405601843 +0000 UTC m=+103.554712121, 107.15608ms after stability
2025-06-26T05:14:35.405Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: aborting pipeline replication: peer="{Voter server-88fe5755-daa0-db68-2267-849ed9724abf 88fe5755-daa0-db68-2267-849ed9724abf}"
2025-06-26T05:14:35.405Z [INFO]  server-7419a4e7-0067-7934-5680-dce6362e028d: aborting pipeline replication: peer="{Voter server-8f1b7583-0405-5c4b-afd9-96a6bc85355d 8f1b7583-0405-5c4b-afd9-96a6bc85355d}"
--- PASS: TestRaft_LeadershipTransfer (0.38s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2025-06-26T05:14:35.406Z [INFO]  server-004ae6f3-9a51-46c6-b6db-2e0019dd55e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-004ae6f3-9a51-46c6-b6db-2e0019dd55e9 Address:004ae6f3-9a51-46c6-b6db-2e0019dd55e9}]"
2025-06-26T05:14:35.406Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:35.406Z [INFO]  server-004ae6f3-9a51-46c6-b6db-2e0019dd55e9: entering follower state: follower="Node at 004ae6f3-9a51-46c6-b6db-2e0019dd55e9 [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.505Z [WARN]  server-004ae6f3-9a51-46c6-b6db-2e0019dd55e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:35.505Z [INFO]  server-004ae6f3-9a51-46c6-b6db-2e0019dd55e9: entering candidate state: node="Node at 004ae6f3-9a51-46c6-b6db-2e0019dd55e9 [Candidate]" term=2
2025-06-26T05:14:35.505Z [INFO]  server-004ae6f3-9a51-46c6-b6db-2e0019dd55e9: election won: term=2 tally=1
2025-06-26T05:14:35.505Z [INFO]  server-004ae6f3-9a51-46c6-b6db-2e0019dd55e9: entering leader state: leader="Node at 004ae6f3-9a51-46c6-b6db-2e0019dd55e9 [Leader]"
2025-06-26T05:14:35.613Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:35.506129203 +0000 UTC m=+103.655239481 (1 nodes), 99.90788ms from start of poll, 100.0132ms from cluster start. Timeout at 2025-06-26 05:14:35.613600443 +0000 UTC m=+103.762710681, 107.4712ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.21s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2025-06-26T05:14:35.614Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb686f86-8dab-8a18-b33f-5f7d9a688110 Address:fb686f86-8dab-8a18-b33f-5f7d9a688110} {Suffrage:Voter ID:server-18a4423d-6e30-20cc-02ae-a9a054de1a6e Address:18a4423d-6e30-20cc-02ae-a9a054de1a6e} {Suffrage:Voter ID:server-91402d23-733b-1720-3e78-684b871f3128 Address:91402d23-733b-1720-3e78-684b871f3128} {Suffrage:Voter ID:server-7fd49b8d-d61d-a2f7-638b-52043c72f37f Address:7fd49b8d-d61d-a2f7-638b-52043c72f37f} {Suffrage:Voter ID:server-fa62ecb7-7390-0a98-832e-79ca5a065e05 Address:fa62ecb7-7390-0a98-832e-79ca5a065e05} {Suffrage:Voter ID:server-9669c77d-d650-dfaf-6293-c68790f2b115 Address:9669c77d-d650-dfaf-6293-c68790f2b115} {Suffrage:Voter ID:server-ea963769-9721-e12f-4598-5bc97b43b719 Address:ea963769-9721-e12f-4598-5bc97b43b719}]"
2025-06-26T05:14:35.614Z [INFO]  server-18a4423d-6e30-20cc-02ae-a9a054de1a6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb686f86-8dab-8a18-b33f-5f7d9a688110 Address:fb686f86-8dab-8a18-b33f-5f7d9a688110} {Suffrage:Voter ID:server-18a4423d-6e30-20cc-02ae-a9a054de1a6e Address:18a4423d-6e30-20cc-02ae-a9a054de1a6e} {Suffrage:Voter ID:server-91402d23-733b-1720-3e78-684b871f3128 Address:91402d23-733b-1720-3e78-684b871f3128} {Suffrage:Voter ID:server-7fd49b8d-d61d-a2f7-638b-52043c72f37f Address:7fd49b8d-d61d-a2f7-638b-52043c72f37f} {Suffrage:Voter ID:server-fa62ecb7-7390-0a98-832e-79ca5a065e05 Address:fa62ecb7-7390-0a98-832e-79ca5a065e05} {Suffrage:Voter ID:server-9669c77d-d650-dfaf-6293-c68790f2b115 Address:9669c77d-d650-dfaf-6293-c68790f2b115} {Suffrage:Voter ID:server-ea963769-9721-e12f-4598-5bc97b43b719 Address:ea963769-9721-e12f-4598-5bc97b43b719}]"
2025-06-26T05:14:35.614Z [INFO]  server-91402d23-733b-1720-3e78-684b871f3128: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb686f86-8dab-8a18-b33f-5f7d9a688110 Address:fb686f86-8dab-8a18-b33f-5f7d9a688110} {Suffrage:Voter ID:server-18a4423d-6e30-20cc-02ae-a9a054de1a6e Address:18a4423d-6e30-20cc-02ae-a9a054de1a6e} {Suffrage:Voter ID:server-91402d23-733b-1720-3e78-684b871f3128 Address:91402d23-733b-1720-3e78-684b871f3128} {Suffrage:Voter ID:server-7fd49b8d-d61d-a2f7-638b-52043c72f37f Address:7fd49b8d-d61d-a2f7-638b-52043c72f37f} {Suffrage:Voter ID:server-fa62ecb7-7390-0a98-832e-79ca5a065e05 Address:fa62ecb7-7390-0a98-832e-79ca5a065e05} {Suffrage:Voter ID:server-9669c77d-d650-dfaf-6293-c68790f2b115 Address:9669c77d-d650-dfaf-6293-c68790f2b115} {Suffrage:Voter ID:server-ea963769-9721-e12f-4598-5bc97b43b719 Address:ea963769-9721-e12f-4598-5bc97b43b719}]"
2025-06-26T05:14:35.614Z [INFO]  server-7fd49b8d-d61d-a2f7-638b-52043c72f37f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb686f86-8dab-8a18-b33f-5f7d9a688110 Address:fb686f86-8dab-8a18-b33f-5f7d9a688110} {Suffrage:Voter ID:server-18a4423d-6e30-20cc-02ae-a9a054de1a6e Address:18a4423d-6e30-20cc-02ae-a9a054de1a6e} {Suffrage:Voter ID:server-91402d23-733b-1720-3e78-684b871f3128 Address:91402d23-733b-1720-3e78-684b871f3128} {Suffrage:Voter ID:server-7fd49b8d-d61d-a2f7-638b-52043c72f37f Address:7fd49b8d-d61d-a2f7-638b-52043c72f37f} {Suffrage:Voter ID:server-fa62ecb7-7390-0a98-832e-79ca5a065e05 Address:fa62ecb7-7390-0a98-832e-79ca5a065e05} {Suffrage:Voter ID:server-9669c77d-d650-dfaf-6293-c68790f2b115 Address:9669c77d-d650-dfaf-6293-c68790f2b115} {Suffrage:Voter ID:server-ea963769-9721-e12f-4598-5bc97b43b719 Address:ea963769-9721-e12f-4598-5bc97b43b719}]"
2025-06-26T05:14:35.614Z [INFO]  server-fa62ecb7-7390-0a98-832e-79ca5a065e05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb686f86-8dab-8a18-b33f-5f7d9a688110 Address:fb686f86-8dab-8a18-b33f-5f7d9a688110} {Suffrage:Voter ID:server-18a4423d-6e30-20cc-02ae-a9a054de1a6e Address:18a4423d-6e30-20cc-02ae-a9a054de1a6e} {Suffrage:Voter ID:server-91402d23-733b-1720-3e78-684b871f3128 Address:91402d23-733b-1720-3e78-684b871f3128} {Suffrage:Voter ID:server-7fd49b8d-d61d-a2f7-638b-52043c72f37f Address:7fd49b8d-d61d-a2f7-638b-52043c72f37f} {Suffrage:Voter ID:server-fa62ecb7-7390-0a98-832e-79ca5a065e05 Address:fa62ecb7-7390-0a98-832e-79ca5a065e05} {Suffrage:Voter ID:server-9669c77d-d650-dfaf-6293-c68790f2b115 Address:9669c77d-d650-dfaf-6293-c68790f2b115} {Suffrage:Voter ID:server-ea963769-9721-e12f-4598-5bc97b43b719 Address:ea963769-9721-e12f-4598-5bc97b43b719}]"
2025-06-26T05:14:35.614Z [INFO]  server-9669c77d-d650-dfaf-6293-c68790f2b115: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb686f86-8dab-8a18-b33f-5f7d9a688110 Address:fb686f86-8dab-8a18-b33f-5f7d9a688110} {Suffrage:Voter ID:server-18a4423d-6e30-20cc-02ae-a9a054de1a6e Address:18a4423d-6e30-20cc-02ae-a9a054de1a6e} {Suffrage:Voter ID:server-91402d23-733b-1720-3e78-684b871f3128 Address:91402d23-733b-1720-3e78-684b871f3128} {Suffrage:Voter ID:server-7fd49b8d-d61d-a2f7-638b-52043c72f37f Address:7fd49b8d-d61d-a2f7-638b-52043c72f37f} {Suffrage:Voter ID:server-fa62ecb7-7390-0a98-832e-79ca5a065e05 Address:fa62ecb7-7390-0a98-832e-79ca5a065e05} {Suffrage:Voter ID:server-9669c77d-d650-dfaf-6293-c68790f2b115 Address:9669c77d-d650-dfaf-6293-c68790f2b115} {Suffrage:Voter ID:server-ea963769-9721-e12f-4598-5bc97b43b719 Address:ea963769-9721-e12f-4598-5bc97b43b719}]"
2025-06-26T05:14:35.614Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb686f86-8dab-8a18-b33f-5f7d9a688110 Address:fb686f86-8dab-8a18-b33f-5f7d9a688110} {Suffrage:Voter ID:server-18a4423d-6e30-20cc-02ae-a9a054de1a6e Address:18a4423d-6e30-20cc-02ae-a9a054de1a6e} {Suffrage:Voter ID:server-91402d23-733b-1720-3e78-684b871f3128 Address:91402d23-733b-1720-3e78-684b871f3128} {Suffrage:Voter ID:server-7fd49b8d-d61d-a2f7-638b-52043c72f37f Address:7fd49b8d-d61d-a2f7-638b-52043c72f37f} {Suffrage:Voter ID:server-fa62ecb7-7390-0a98-832e-79ca5a065e05 Address:fa62ecb7-7390-0a98-832e-79ca5a065e05} {Suffrage:Voter ID:server-9669c77d-d650-dfaf-6293-c68790f2b115 Address:9669c77d-d650-dfaf-6293-c68790f2b115} {Suffrage:Voter ID:server-ea963769-9721-e12f-4598-5bc97b43b719 Address:ea963769-9721-e12f-4598-5bc97b43b719}]"
2025-06-26T05:14:35.614Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:35.614Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: entering follower state: follower="Node at fb686f86-8dab-8a18-b33f-5f7d9a688110 [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.615Z [INFO]  server-18a4423d-6e30-20cc-02ae-a9a054de1a6e: entering follower state: follower="Node at 18a4423d-6e30-20cc-02ae-a9a054de1a6e [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.615Z [INFO]  server-91402d23-733b-1720-3e78-684b871f3128: entering follower state: follower="Node at 91402d23-733b-1720-3e78-684b871f3128 [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.615Z [INFO]  server-7fd49b8d-d61d-a2f7-638b-52043c72f37f: entering follower state: follower="Node at 7fd49b8d-d61d-a2f7-638b-52043c72f37f [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.615Z [INFO]  server-fa62ecb7-7390-0a98-832e-79ca5a065e05: entering follower state: follower="Node at fa62ecb7-7390-0a98-832e-79ca5a065e05 [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.615Z [INFO]  server-9669c77d-d650-dfaf-6293-c68790f2b115: entering follower state: follower="Node at 9669c77d-d650-dfaf-6293-c68790f2b115 [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.615Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: entering follower state: follower="Node at ea963769-9721-e12f-4598-5bc97b43b719 [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.669Z [WARN]  server-ea963769-9721-e12f-4598-5bc97b43b719: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: entering candidate state: node="Node at ea963769-9721-e12f-4598-5bc97b43b719 [Candidate]" term=2
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: election won: term=2 tally=4
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: entering leader state: leader="Node at ea963769-9721-e12f-4598-5bc97b43b719 [Leader]"
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: added peer, starting replication: peer=server-fb686f86-8dab-8a18-b33f-5f7d9a688110
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: added peer, starting replication: peer=server-18a4423d-6e30-20cc-02ae-a9a054de1a6e
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: added peer, starting replication: peer=server-91402d23-733b-1720-3e78-684b871f3128
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: added peer, starting replication: peer=server-7fd49b8d-d61d-a2f7-638b-52043c72f37f
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: added peer, starting replication: peer=server-fa62ecb7-7390-0a98-832e-79ca5a065e05
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: added peer, starting replication: peer=server-9669c77d-d650-dfaf-6293-c68790f2b115
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: pipelining replication: peer="{Voter server-9669c77d-d650-dfaf-6293-c68790f2b115 9669c77d-d650-dfaf-6293-c68790f2b115}"
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: pipelining replication: peer="{Voter server-fb686f86-8dab-8a18-b33f-5f7d9a688110 fb686f86-8dab-8a18-b33f-5f7d9a688110}"
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: pipelining replication: peer="{Voter server-18a4423d-6e30-20cc-02ae-a9a054de1a6e 18a4423d-6e30-20cc-02ae-a9a054de1a6e}"
2025-06-26T05:14:35.669Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: pipelining replication: peer="{Voter server-7fd49b8d-d61d-a2f7-638b-52043c72f37f 7fd49b8d-d61d-a2f7-638b-52043c72f37f}"
2025-06-26T05:14:35.670Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: pipelining replication: peer="{Voter server-fa62ecb7-7390-0a98-832e-79ca5a065e05 fa62ecb7-7390-0a98-832e-79ca5a065e05}"
2025-06-26T05:14:35.670Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: pipelining replication: peer="{Voter server-91402d23-733b-1720-3e78-684b871f3128 91402d23-733b-1720-3e78-684b871f3128}"
2025-06-26T05:14:35.777Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:35.669816883 +0000 UTC m=+103.818927161 (1 nodes), 54.94268ms from start of poll, 55.4426ms from cluster start. Timeout at 2025-06-26 05:14:35.777642843 +0000 UTC m=+103.926753121, 107.82596ms after stability
2025-06-26T05:14:35.777Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:35.882Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:35.777806803 +0000 UTC m=+103.926917081 (6 nodes), 2.84µs from start of poll, 163.43252ms from cluster start. Timeout at 2025-06-26 05:14:35.882943803 +0000 UTC m=+104.032054081, 105.137ms after stability
2025-06-26T05:14:35.883Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:35.989Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:35.883030163 +0000 UTC m=+104.032140441 (1 nodes), 2.4µs from start of poll, 268.65588ms from cluster start. Timeout at 2025-06-26 05:14:35.989608843 +0000 UTC m=+104.138719121, 106.57868ms after stability
2025-06-26T05:14:35.989Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: entering candidate state: node="Node at fb686f86-8dab-8a18-b33f-5f7d9a688110 [Candidate]" term=3
2025-06-26T05:14:35.989Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: entering follower state: follower="Node at ea963769-9721-e12f-4598-5bc97b43b719 [Follower]" leader-address= leader-id=
2025-06-26T05:14:35.989Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: aborting pipeline replication: peer="{Voter server-9669c77d-d650-dfaf-6293-c68790f2b115 9669c77d-d650-dfaf-6293-c68790f2b115}"
2025-06-26T05:14:35.990Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: election won: term=3 tally=4
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: entering leader state: leader="Node at fb686f86-8dab-8a18-b33f-5f7d9a688110 [Leader]"
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: added peer, starting replication: peer=server-18a4423d-6e30-20cc-02ae-a9a054de1a6e
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: added peer, starting replication: peer=server-91402d23-733b-1720-3e78-684b871f3128
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: added peer, starting replication: peer=server-7fd49b8d-d61d-a2f7-638b-52043c72f37f
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: added peer, starting replication: peer=server-fa62ecb7-7390-0a98-832e-79ca5a065e05
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: added peer, starting replication: peer=server-9669c77d-d650-dfaf-6293-c68790f2b115
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: added peer, starting replication: peer=server-ea963769-9721-e12f-4598-5bc97b43b719
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: pipelining replication: peer="{Voter server-ea963769-9721-e12f-4598-5bc97b43b719 ea963769-9721-e12f-4598-5bc97b43b719}"
2025-06-26T05:14:35.990Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: aborting pipeline replication: peer="{Voter server-fb686f86-8dab-8a18-b33f-5f7d9a688110 fb686f86-8dab-8a18-b33f-5f7d9a688110}"
2025-06-26T05:14:35.990Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: aborting pipeline replication: peer="{Voter server-18a4423d-6e30-20cc-02ae-a9a054de1a6e 18a4423d-6e30-20cc-02ae-a9a054de1a6e}"
2025-06-26T05:14:35.990Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: aborting pipeline replication: peer="{Voter server-91402d23-733b-1720-3e78-684b871f3128 91402d23-733b-1720-3e78-684b871f3128}"
2025-06-26T05:14:35.990Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: aborting pipeline replication: peer="{Voter server-7fd49b8d-d61d-a2f7-638b-52043c72f37f 7fd49b8d-d61d-a2f7-638b-52043c72f37f}"
2025-06-26T05:14:35.990Z [INFO]  server-ea963769-9721-e12f-4598-5bc97b43b719: aborting pipeline replication: peer="{Voter server-fa62ecb7-7390-0a98-832e-79ca5a065e05 fa62ecb7-7390-0a98-832e-79ca5a065e05}"
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: pipelining replication: peer="{Voter server-18a4423d-6e30-20cc-02ae-a9a054de1a6e 18a4423d-6e30-20cc-02ae-a9a054de1a6e}"
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: pipelining replication: peer="{Voter server-91402d23-733b-1720-3e78-684b871f3128 91402d23-733b-1720-3e78-684b871f3128}"
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: pipelining replication: peer="{Voter server-7fd49b8d-d61d-a2f7-638b-52043c72f37f 7fd49b8d-d61d-a2f7-638b-52043c72f37f}"
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: pipelining replication: peer="{Voter server-fa62ecb7-7390-0a98-832e-79ca5a065e05 fa62ecb7-7390-0a98-832e-79ca5a065e05}"
2025-06-26T05:14:35.990Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: pipelining replication: peer="{Voter server-9669c77d-d650-dfaf-6293-c68790f2b115 9669c77d-d650-dfaf-6293-c68790f2b115}"
2025-06-26T05:14:36.101Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:35.990308163 +0000 UTC m=+104.139418441 (1 nodes), 272.52µs from start of poll, 375.93388ms from cluster start. Timeout at 2025-06-26 05:14:36.101719283 +0000 UTC m=+104.250829561, 111.41112ms after stability
2025-06-26T05:14:36.101Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: aborting pipeline replication: peer="{Voter server-ea963769-9721-e12f-4598-5bc97b43b719 ea963769-9721-e12f-4598-5bc97b43b719}"
2025-06-26T05:14:36.102Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: aborting pipeline replication: peer="{Voter server-18a4423d-6e30-20cc-02ae-a9a054de1a6e 18a4423d-6e30-20cc-02ae-a9a054de1a6e}"
2025-06-26T05:14:36.102Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: aborting pipeline replication: peer="{Voter server-fa62ecb7-7390-0a98-832e-79ca5a065e05 fa62ecb7-7390-0a98-832e-79ca5a065e05}"
2025-06-26T05:14:36.102Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: aborting pipeline replication: peer="{Voter server-91402d23-733b-1720-3e78-684b871f3128 91402d23-733b-1720-3e78-684b871f3128}"
2025-06-26T05:14:36.102Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: aborting pipeline replication: peer="{Voter server-7fd49b8d-d61d-a2f7-638b-52043c72f37f 7fd49b8d-d61d-a2f7-638b-52043c72f37f}"
2025-06-26T05:14:36.102Z [INFO]  server-fb686f86-8dab-8a18-b33f-5f7d9a688110: aborting pipeline replication: peer="{Voter server-9669c77d-d650-dfaf-6293-c68790f2b115 9669c77d-d650-dfaf-6293-c68790f2b115}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2025-06-26T05:14:36.102Z [INFO]  server-3bdadc4a-20c0-1039-4656-e26726e374dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bdadc4a-20c0-1039-4656-e26726e374dc Address:3bdadc4a-20c0-1039-4656-e26726e374dc} {Suffrage:Voter ID:server-1915b894-34ef-ad3e-e610-f5e490d48c17 Address:1915b894-34ef-ad3e-e610-f5e490d48c17} {Suffrage:Voter ID:server-6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec Address:6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec}]"
2025-06-26T05:14:36.102Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bdadc4a-20c0-1039-4656-e26726e374dc Address:3bdadc4a-20c0-1039-4656-e26726e374dc} {Suffrage:Voter ID:server-1915b894-34ef-ad3e-e610-f5e490d48c17 Address:1915b894-34ef-ad3e-e610-f5e490d48c17} {Suffrage:Voter ID:server-6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec Address:6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec}]"
2025-06-26T05:14:36.102Z [INFO]  server-6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bdadc4a-20c0-1039-4656-e26726e374dc Address:3bdadc4a-20c0-1039-4656-e26726e374dc} {Suffrage:Voter ID:server-1915b894-34ef-ad3e-e610-f5e490d48c17 Address:1915b894-34ef-ad3e-e610-f5e490d48c17} {Suffrage:Voter ID:server-6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec Address:6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec}]"
2025-06-26T05:14:36.102Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:36.102Z [INFO]  server-3bdadc4a-20c0-1039-4656-e26726e374dc: entering follower state: follower="Node at 3bdadc4a-20c0-1039-4656-e26726e374dc [Follower]" leader-address= leader-id=
2025-06-26T05:14:36.102Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: entering follower state: follower="Node at 1915b894-34ef-ad3e-e610-f5e490d48c17 [Follower]" leader-address= leader-id=
2025-06-26T05:14:36.102Z [INFO]  server-6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec: entering follower state: follower="Node at 6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec [Follower]" leader-address= leader-id=
2025-06-26T05:14:36.185Z [WARN]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:36.185Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: entering candidate state: node="Node at 1915b894-34ef-ad3e-e610-f5e490d48c17 [Candidate]" term=2
2025-06-26T05:14:36.185Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: election won: term=2 tally=2
2025-06-26T05:14:36.185Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: entering leader state: leader="Node at 1915b894-34ef-ad3e-e610-f5e490d48c17 [Leader]"
2025-06-26T05:14:36.185Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: added peer, starting replication: peer=server-3bdadc4a-20c0-1039-4656-e26726e374dc
2025-06-26T05:14:36.185Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: added peer, starting replication: peer=server-6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec
2025-06-26T05:14:36.185Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: pipelining replication: peer="{Voter server-6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec 6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec}"
2025-06-26T05:14:36.185Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: pipelining replication: peer="{Voter server-3bdadc4a-20c0-1039-4656-e26726e374dc 3bdadc4a-20c0-1039-4656-e26726e374dc}"
2025-06-26T05:14:36.185Z [WARN]  server-3bdadc4a-20c0-1039-4656-e26726e374dc: rejecting vote request since our last term is greater: candidate=1915b894-34ef-ad3e-e610-f5e490d48c17 last-term=2 last-candidate-term=1
2025-06-26T05:14:36.293Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:36.186006683 +0000 UTC m=+104.335116961 (1 nodes), 83.1824ms from start of poll, 83.38416ms from cluster start. Timeout at 2025-06-26 05:14:36.293605843 +0000 UTC m=+104.442716081, 107.59912ms after stability
2025-06-26T05:14:36.293Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: aborting pipeline replication: peer="{Voter server-6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec 6802cdef-a0a9-eef1-4fa8-4ef8d7ceeaec}"
2025-06-26T05:14:36.293Z [INFO]  server-1915b894-34ef-ad3e-e610-f5e490d48c17: aborting pipeline replication: peer="{Voter server-3bdadc4a-20c0-1039-4656-e26726e374dc 3bdadc4a-20c0-1039-4656-e26726e374dc}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.19s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2025-06-26T05:14:36.294Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38e33d8a-71ca-2525-f42d-25aea835d30e Address:38e33d8a-71ca-2525-f42d-25aea835d30e} {Suffrage:Voter ID:server-b24d2cda-a125-a5c1-89aa-eeee364441a8 Address:b24d2cda-a125-a5c1-89aa-eeee364441a8} {Suffrage:Voter ID:server-c891de6b-cdba-8490-eece-a4b112c1e405 Address:c891de6b-cdba-8490-eece-a4b112c1e405}]"
2025-06-26T05:14:36.294Z [INFO]  server-b24d2cda-a125-a5c1-89aa-eeee364441a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38e33d8a-71ca-2525-f42d-25aea835d30e Address:38e33d8a-71ca-2525-f42d-25aea835d30e} {Suffrage:Voter ID:server-b24d2cda-a125-a5c1-89aa-eeee364441a8 Address:b24d2cda-a125-a5c1-89aa-eeee364441a8} {Suffrage:Voter ID:server-c891de6b-cdba-8490-eece-a4b112c1e405 Address:c891de6b-cdba-8490-eece-a4b112c1e405}]"
2025-06-26T05:14:36.294Z [INFO]  server-c891de6b-cdba-8490-eece-a4b112c1e405: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38e33d8a-71ca-2525-f42d-25aea835d30e Address:38e33d8a-71ca-2525-f42d-25aea835d30e} {Suffrage:Voter ID:server-b24d2cda-a125-a5c1-89aa-eeee364441a8 Address:b24d2cda-a125-a5c1-89aa-eeee364441a8} {Suffrage:Voter ID:server-c891de6b-cdba-8490-eece-a4b112c1e405 Address:c891de6b-cdba-8490-eece-a4b112c1e405}]"
2025-06-26T05:14:36.294Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:36.294Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: entering follower state: follower="Node at 38e33d8a-71ca-2525-f42d-25aea835d30e [Follower]" leader-address= leader-id=
2025-06-26T05:14:36.294Z [INFO]  server-b24d2cda-a125-a5c1-89aa-eeee364441a8: entering follower state: follower="Node at b24d2cda-a125-a5c1-89aa-eeee364441a8 [Follower]" leader-address= leader-id=
2025-06-26T05:14:36.294Z [INFO]  server-c891de6b-cdba-8490-eece-a4b112c1e405: entering follower state: follower="Node at c891de6b-cdba-8490-eece-a4b112c1e405 [Follower]" leader-address= leader-id=
2025-06-26T05:14:36.369Z [WARN]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:36.369Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: entering candidate state: node="Node at 38e33d8a-71ca-2525-f42d-25aea835d30e [Candidate]" term=2
2025-06-26T05:14:36.369Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: election won: term=2 tally=2
2025-06-26T05:14:36.369Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: entering leader state: leader="Node at 38e33d8a-71ca-2525-f42d-25aea835d30e [Leader]"
2025-06-26T05:14:36.369Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: added peer, starting replication: peer=server-b24d2cda-a125-a5c1-89aa-eeee364441a8
2025-06-26T05:14:36.369Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: added peer, starting replication: peer=server-c891de6b-cdba-8490-eece-a4b112c1e405
2025-06-26T05:14:36.369Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: pipelining replication: peer="{Voter server-c891de6b-cdba-8490-eece-a4b112c1e405 c891de6b-cdba-8490-eece-a4b112c1e405}"
2025-06-26T05:14:36.370Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: pipelining replication: peer="{Voter server-b24d2cda-a125-a5c1-89aa-eeee364441a8 b24d2cda-a125-a5c1-89aa-eeee364441a8}"
2025-06-26T05:14:36.481Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:36.370165883 +0000 UTC m=+104.519276121 (2 nodes), 75.6114ms from start of poll, 75.82852ms from cluster start. Timeout at 2025-06-26 05:14:36.481762963 +0000 UTC m=+104.630873201, 111.59708ms after stability
2025-06-26T05:14:36.481Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:36.589Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:36.481832323 +0000 UTC m=+104.630942601 (1 nodes), 1.8µs from start of poll, 187.495ms from cluster start. Timeout at 2025-06-26 05:14:36.589801443 +0000 UTC m=+104.738911721, 107.96912ms after stability
2025-06-26T05:14:36.589Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: aborting pipeline replication: peer="{Voter server-c891de6b-cdba-8490-eece-a4b112c1e405 c891de6b-cdba-8490-eece-a4b112c1e405}"
2025-06-26T05:14:36.589Z [INFO]  server-38e33d8a-71ca-2525-f42d-25aea835d30e: aborting pipeline replication: peer="{Voter server-b24d2cda-a125-a5c1-89aa-eeee364441a8 b24d2cda-a125-a5c1-89aa-eeee364441a8}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.30s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2025-06-26T05:14:36.590Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f560898b-f328-803a-733f-6d2c0bb66490 Address:f560898b-f328-803a-733f-6d2c0bb66490} {Suffrage:Voter ID:server-9ad09646-63da-686a-c3f0-1fd5d9be66de Address:9ad09646-63da-686a-c3f0-1fd5d9be66de} {Suffrage:Voter ID:server-f7a1307e-f6ab-b324-a8df-456f33d1cc75 Address:f7a1307e-f6ab-b324-a8df-456f33d1cc75}]"
2025-06-26T05:14:36.590Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f560898b-f328-803a-733f-6d2c0bb66490 Address:f560898b-f328-803a-733f-6d2c0bb66490} {Suffrage:Voter ID:server-9ad09646-63da-686a-c3f0-1fd5d9be66de Address:9ad09646-63da-686a-c3f0-1fd5d9be66de} {Suffrage:Voter ID:server-f7a1307e-f6ab-b324-a8df-456f33d1cc75 Address:f7a1307e-f6ab-b324-a8df-456f33d1cc75}]"
2025-06-26T05:14:36.590Z [INFO]  server-f7a1307e-f6ab-b324-a8df-456f33d1cc75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f560898b-f328-803a-733f-6d2c0bb66490 Address:f560898b-f328-803a-733f-6d2c0bb66490} {Suffrage:Voter ID:server-9ad09646-63da-686a-c3f0-1fd5d9be66de Address:9ad09646-63da-686a-c3f0-1fd5d9be66de} {Suffrage:Voter ID:server-f7a1307e-f6ab-b324-a8df-456f33d1cc75 Address:f7a1307e-f6ab-b324-a8df-456f33d1cc75}]"
2025-06-26T05:14:36.590Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:36.590Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: entering follower state: follower="Node at f560898b-f328-803a-733f-6d2c0bb66490 [Follower]" leader-address= leader-id=
2025-06-26T05:14:36.590Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: entering follower state: follower="Node at 9ad09646-63da-686a-c3f0-1fd5d9be66de [Follower]" leader-address= leader-id=
2025-06-26T05:14:36.590Z [INFO]  server-f7a1307e-f6ab-b324-a8df-456f33d1cc75: entering follower state: follower="Node at f7a1307e-f6ab-b324-a8df-456f33d1cc75 [Follower]" leader-address= leader-id=
2025-06-26T05:14:36.653Z [WARN]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:36.653Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: entering candidate state: node="Node at 9ad09646-63da-686a-c3f0-1fd5d9be66de [Candidate]" term=2
2025-06-26T05:14:36.653Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: election won: term=2 tally=2
2025-06-26T05:14:36.653Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: entering leader state: leader="Node at 9ad09646-63da-686a-c3f0-1fd5d9be66de [Leader]"
2025-06-26T05:14:36.653Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: added peer, starting replication: peer=server-f560898b-f328-803a-733f-6d2c0bb66490
2025-06-26T05:14:36.653Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: added peer, starting replication: peer=server-f7a1307e-f6ab-b324-a8df-456f33d1cc75
2025-06-26T05:14:36.653Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: pipelining replication: peer="{Voter server-f7a1307e-f6ab-b324-a8df-456f33d1cc75 f7a1307e-f6ab-b324-a8df-456f33d1cc75}"
2025-06-26T05:14:36.653Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: pipelining replication: peer="{Voter server-f560898b-f328-803a-733f-6d2c0bb66490 f560898b-f328-803a-733f-6d2c0bb66490}"
2025-06-26T05:14:36.761Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:36.653958483 +0000 UTC m=+104.803068761 (1 nodes), 63.51404ms from start of poll, 63.703ms from cluster start. Timeout at 2025-06-26 05:14:36.761604683 +0000 UTC m=+104.910714961, 107.6462ms after stability
2025-06-26T05:14:36.761Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:36.870Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:36.761728203 +0000 UTC m=+104.910838441 (2 nodes), 1.84µs from start of poll, 171.47268ms from cluster start. Timeout at 2025-06-26 05:14:36.870009523 +0000 UTC m=+105.019119801, 108.28136ms after stability
2025-06-26T05:14:36.919Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: entering candidate state: node="Node at f560898b-f328-803a-733f-6d2c0bb66490 [Candidate]" term=3
2025-06-26T05:14:36.919Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: election won: term=3 tally=2
2025-06-26T05:14:36.919Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: entering leader state: leader="Node at f560898b-f328-803a-733f-6d2c0bb66490 [Leader]"
2025-06-26T05:14:36.919Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: added peer, starting replication: peer=server-9ad09646-63da-686a-c3f0-1fd5d9be66de
2025-06-26T05:14:36.919Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: added peer, starting replication: peer=server-f7a1307e-f6ab-b324-a8df-456f33d1cc75
2025-06-26T05:14:36.919Z [WARN]  server-f7a1307e-f6ab-b324-a8df-456f33d1cc75: failed to get previous log: previous-index=1002 last-index=778 error="log not found"
2025-06-26T05:14:36.919Z [WARN]  server-f560898b-f328-803a-733f-6d2c0bb66490: appendEntries rejected, sending older logs: peer="{Voter server-f7a1307e-f6ab-b324-a8df-456f33d1cc75 f7a1307e-f6ab-b324-a8df-456f33d1cc75}" next=779
2025-06-26T05:14:36.919Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: pipelining replication: peer="{Voter server-f7a1307e-f6ab-b324-a8df-456f33d1cc75 f7a1307e-f6ab-b324-a8df-456f33d1cc75}"
2025-06-26T05:14:36.919Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:36.919Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: entering follower state: follower="Node at 9ad09646-63da-686a-c3f0-1fd5d9be66de [Follower]" leader-address= leader-id=
2025-06-26T05:14:36.919Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: aborting pipeline replication: peer="{Voter server-f7a1307e-f6ab-b324-a8df-456f33d1cc75 f7a1307e-f6ab-b324-a8df-456f33d1cc75}"
2025-06-26T05:14:36.919Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: pipelining replication: peer="{Voter server-9ad09646-63da-686a-c3f0-1fd5d9be66de 9ad09646-63da-686a-c3f0-1fd5d9be66de}"
2025-06-26T05:14:36.920Z [INFO]  server-9ad09646-63da-686a-c3f0-1fd5d9be66de: aborting pipeline replication: peer="{Voter server-f560898b-f328-803a-733f-6d2c0bb66490 f560898b-f328-803a-733f-6d2c0bb66490}"
2025-06-26T05:14:37.025Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:36.920234443 +0000 UTC m=+105.069344681 (1 nodes), 597µs from start of poll, 329.97892ms from cluster start. Timeout at 2025-06-26 05:14:37.025613963 +0000 UTC m=+105.174724241, 105.37956ms after stability
2025-06-26T05:14:37.025Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: aborting pipeline replication: peer="{Voter server-f7a1307e-f6ab-b324-a8df-456f33d1cc75 f7a1307e-f6ab-b324-a8df-456f33d1cc75}"
2025-06-26T05:14:37.025Z [INFO]  server-f560898b-f328-803a-733f-6d2c0bb66490: aborting pipeline replication: peer="{Voter server-9ad09646-63da-686a-c3f0-1fd5d9be66de 9ad09646-63da-686a-c3f0-1fd5d9be66de}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.44s)
=== RUN   TestRaft_LeadershipTransferToItself
2025-06-26T05:14:37.026Z [INFO]  server-cfd57b66-b9ed-b0c9-9db7-26c5a178192f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfd57b66-b9ed-b0c9-9db7-26c5a178192f Address:cfd57b66-b9ed-b0c9-9db7-26c5a178192f} {Suffrage:Voter ID:server-0ae5a85d-096c-d52d-f5c3-36cca68460eb Address:0ae5a85d-096c-d52d-f5c3-36cca68460eb} {Suffrage:Voter ID:server-9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb Address:9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb}]"
2025-06-26T05:14:37.026Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfd57b66-b9ed-b0c9-9db7-26c5a178192f Address:cfd57b66-b9ed-b0c9-9db7-26c5a178192f} {Suffrage:Voter ID:server-0ae5a85d-096c-d52d-f5c3-36cca68460eb Address:0ae5a85d-096c-d52d-f5c3-36cca68460eb} {Suffrage:Voter ID:server-9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb Address:9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb}]"
2025-06-26T05:14:37.026Z [INFO]  server-9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfd57b66-b9ed-b0c9-9db7-26c5a178192f Address:cfd57b66-b9ed-b0c9-9db7-26c5a178192f} {Suffrage:Voter ID:server-0ae5a85d-096c-d52d-f5c3-36cca68460eb Address:0ae5a85d-096c-d52d-f5c3-36cca68460eb} {Suffrage:Voter ID:server-9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb Address:9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb}]"
2025-06-26T05:14:37.026Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:37.026Z [INFO]  server-cfd57b66-b9ed-b0c9-9db7-26c5a178192f: entering follower state: follower="Node at cfd57b66-b9ed-b0c9-9db7-26c5a178192f [Follower]" leader-address= leader-id=
2025-06-26T05:14:37.026Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: entering follower state: follower="Node at 0ae5a85d-096c-d52d-f5c3-36cca68460eb [Follower]" leader-address= leader-id=
2025-06-26T05:14:37.026Z [INFO]  server-9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb: entering follower state: follower="Node at 9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb [Follower]" leader-address= leader-id=
2025-06-26T05:14:37.081Z [WARN]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:37.081Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: entering candidate state: node="Node at 0ae5a85d-096c-d52d-f5c3-36cca68460eb [Candidate]" term=2
2025-06-26T05:14:37.081Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: election won: term=2 tally=2
2025-06-26T05:14:37.081Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: entering leader state: leader="Node at 0ae5a85d-096c-d52d-f5c3-36cca68460eb [Leader]"
2025-06-26T05:14:37.081Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: added peer, starting replication: peer=server-cfd57b66-b9ed-b0c9-9db7-26c5a178192f
2025-06-26T05:14:37.081Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: added peer, starting replication: peer=server-9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb
2025-06-26T05:14:37.081Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: pipelining replication: peer="{Voter server-9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb 9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb}"
2025-06-26T05:14:37.081Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: pipelining replication: peer="{Voter server-cfd57b66-b9ed-b0c9-9db7-26c5a178192f cfd57b66-b9ed-b0c9-9db7-26c5a178192f}"
2025-06-26T05:14:37.189Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:37.081868443 +0000 UTC m=+105.230978721 (1 nodes), 55.42844ms from start of poll, 55.6334ms from cluster start. Timeout at 2025-06-26 05:14:37.189749083 +0000 UTC m=+105.338859361, 107.88064ms after stability
2025-06-26T05:14:37.189Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: cannot transfer leadership to itself
2025-06-26T05:14:37.189Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: aborting pipeline replication: peer="{Voter server-9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb 9a0c23d6-d1ad-3ba3-9419-6b9943cdabbb}"
2025-06-26T05:14:37.189Z [INFO]  server-0ae5a85d-096c-d52d-f5c3-36cca68460eb: aborting pipeline replication: peer="{Voter server-cfd57b66-b9ed-b0c9-9db7-26c5a178192f cfd57b66-b9ed-b0c9-9db7-26c5a178192f}"
--- PASS: TestRaft_LeadershipTransferToItself (0.16s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2025-06-26T05:14:37.190Z [INFO]  server-cb554b65-746d-4124-63b1-45f013c0a9f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb554b65-746d-4124-63b1-45f013c0a9f9 Address:cb554b65-746d-4124-63b1-45f013c0a9f9} {Suffrage:Voter ID:server-20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628 Address:20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628} {Suffrage:Voter ID:server-3897cb66-dc79-f673-7be7-382be5abd0a1 Address:3897cb66-dc79-f673-7be7-382be5abd0a1}]"
2025-06-26T05:14:37.190Z [INFO]  server-20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb554b65-746d-4124-63b1-45f013c0a9f9 Address:cb554b65-746d-4124-63b1-45f013c0a9f9} {Suffrage:Voter ID:server-20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628 Address:20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628} {Suffrage:Voter ID:server-3897cb66-dc79-f673-7be7-382be5abd0a1 Address:3897cb66-dc79-f673-7be7-382be5abd0a1}]"
2025-06-26T05:14:37.190Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb554b65-746d-4124-63b1-45f013c0a9f9 Address:cb554b65-746d-4124-63b1-45f013c0a9f9} {Suffrage:Voter ID:server-20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628 Address:20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628} {Suffrage:Voter ID:server-3897cb66-dc79-f673-7be7-382be5abd0a1 Address:3897cb66-dc79-f673-7be7-382be5abd0a1}]"
2025-06-26T05:14:37.190Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:37.190Z [INFO]  server-cb554b65-746d-4124-63b1-45f013c0a9f9: entering follower state: follower="Node at cb554b65-746d-4124-63b1-45f013c0a9f9 [Follower]" leader-address= leader-id=
2025-06-26T05:14:37.190Z [INFO]  server-20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628: entering follower state: follower="Node at 20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628 [Follower]" leader-address= leader-id=
2025-06-26T05:14:37.190Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: entering follower state: follower="Node at 3897cb66-dc79-f673-7be7-382be5abd0a1 [Follower]" leader-address= leader-id=
2025-06-26T05:14:37.249Z [WARN]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:37.249Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: entering candidate state: node="Node at 3897cb66-dc79-f673-7be7-382be5abd0a1 [Candidate]" term=2
2025-06-26T05:14:37.249Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: election won: term=2 tally=2
2025-06-26T05:14:37.249Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: entering leader state: leader="Node at 3897cb66-dc79-f673-7be7-382be5abd0a1 [Leader]"
2025-06-26T05:14:37.249Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: added peer, starting replication: peer=server-cb554b65-746d-4124-63b1-45f013c0a9f9
2025-06-26T05:14:37.249Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: added peer, starting replication: peer=server-20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628
2025-06-26T05:14:37.249Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: pipelining replication: peer="{Voter server-20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628 20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628}"
2025-06-26T05:14:37.249Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: pipelining replication: peer="{Voter server-cb554b65-746d-4124-63b1-45f013c0a9f9 cb554b65-746d-4124-63b1-45f013c0a9f9}"
2025-06-26T05:14:37.361Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:37.249806603 +0000 UTC m=+105.398916881 (1 nodes), 59.407ms from start of poll, 59.60392ms from cluster start. Timeout at 2025-06-26 05:14:37.361900203 +0000 UTC m=+105.511010441, 112.09356ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2025-06-26T05:14:37.362Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: aborting pipeline replication: peer="{Voter server-20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628 20a5eb74-2cf8-6e50-7c2b-ca3fcaa89628}"
2025-06-26T05:14:37.362Z [INFO]  server-3897cb66-dc79-f673-7be7-382be5abd0a1: aborting pipeline replication: peer="{Voter server-cb554b65-746d-4124-63b1-45f013c0a9f9 cb554b65-746d-4124-63b1-45f013c0a9f9}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2025-06-26T05:14:37.362Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50f73960-97ba-10ec-6a9d-8b502789ea5f Address:50f73960-97ba-10ec-6a9d-8b502789ea5f} {Suffrage:Voter ID:server-0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0 Address:0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0} {Suffrage:Voter ID:server-5625df00-4b68-4def-ca24-30897234a014 Address:5625df00-4b68-4def-ca24-30897234a014}]"
2025-06-26T05:14:37.362Z [INFO]  server-0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50f73960-97ba-10ec-6a9d-8b502789ea5f Address:50f73960-97ba-10ec-6a9d-8b502789ea5f} {Suffrage:Voter ID:server-0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0 Address:0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0} {Suffrage:Voter ID:server-5625df00-4b68-4def-ca24-30897234a014 Address:5625df00-4b68-4def-ca24-30897234a014}]"
2025-06-26T05:14:37.362Z [INFO]  server-5625df00-4b68-4def-ca24-30897234a014: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50f73960-97ba-10ec-6a9d-8b502789ea5f Address:50f73960-97ba-10ec-6a9d-8b502789ea5f} {Suffrage:Voter ID:server-0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0 Address:0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0} {Suffrage:Voter ID:server-5625df00-4b68-4def-ca24-30897234a014 Address:5625df00-4b68-4def-ca24-30897234a014}]"
2025-06-26T05:14:37.362Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:37.362Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: entering follower state: follower="Node at 50f73960-97ba-10ec-6a9d-8b502789ea5f [Follower]" leader-address= leader-id=
2025-06-26T05:14:37.362Z [INFO]  server-0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0: entering follower state: follower="Node at 0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0 [Follower]" leader-address= leader-id=
2025-06-26T05:14:37.362Z [INFO]  server-5625df00-4b68-4def-ca24-30897234a014: entering follower state: follower="Node at 5625df00-4b68-4def-ca24-30897234a014 [Follower]" leader-address= leader-id=
2025-06-26T05:14:37.450Z [WARN]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:37.450Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: entering candidate state: node="Node at 50f73960-97ba-10ec-6a9d-8b502789ea5f [Candidate]" term=2
2025-06-26T05:14:37.450Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: election won: term=2 tally=2
2025-06-26T05:14:37.450Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: entering leader state: leader="Node at 50f73960-97ba-10ec-6a9d-8b502789ea5f [Leader]"
2025-06-26T05:14:37.450Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: added peer, starting replication: peer=server-0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0
2025-06-26T05:14:37.450Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: added peer, starting replication: peer=server-5625df00-4b68-4def-ca24-30897234a014
2025-06-26T05:14:37.450Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: pipelining replication: peer="{Voter server-5625df00-4b68-4def-ca24-30897234a014 5625df00-4b68-4def-ca24-30897234a014}"
2025-06-26T05:14:37.450Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: pipelining replication: peer="{Voter server-0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0 0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0}"
2025-06-26T05:14:37.573Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:37.450560563 +0000 UTC m=+105.599670841 (1 nodes), 87.9092ms from start of poll, 88.12972ms from cluster start. Timeout at 2025-06-26 05:14:37.573575043 +0000 UTC m=+105.722685321, 123.01448ms after stability
2025-06-26T05:14:37.573Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:37.685Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:37.573906003 +0000 UTC m=+105.723016281 (2 nodes), 2.24µs from start of poll, 211.47516ms from cluster start. Timeout at 2025-06-26 05:14:37.685568203 +0000 UTC m=+105.834678481, 111.6622ms after stability
2025-06-26T05:14:38.352Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: aborting pipeline replication: peer="{Voter server-5625df00-4b68-4def-ca24-30897234a014 5625df00-4b68-4def-ca24-30897234a014}"
2025-06-26T05:14:38.352Z [INFO]  server-50f73960-97ba-10ec-6a9d-8b502789ea5f: aborting pipeline replication: peer="{Voter server-0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0 0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0}"
2025-06-26T05:14:38.853Z [ERROR] server-50f73960-97ba-10ec-6a9d-8b502789ea5f: failed to heartbeat to: peer=0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0 backoff time=10ms error="command timed out"
2025-06-26T05:14:38.853Z [ERROR] server-50f73960-97ba-10ec-6a9d-8b502789ea5f: failed to appendEntries to: peer="{Voter server-0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0 0d0ecb55-5e73-33ae-cc1d-e1c8f5a2a8c0}" error="command timed out"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (1.49s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2025-06-26T05:14:38.854Z [INFO]  server-c66f9aba-974a-5805-e7ca-1c97ac7934f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c66f9aba-974a-5805-e7ca-1c97ac7934f6 Address:c66f9aba-974a-5805-e7ca-1c97ac7934f6} {Suffrage:Voter ID:server-5c8dc570-440b-fc4f-6de4-8893a772e583 Address:5c8dc570-440b-fc4f-6de4-8893a772e583}]"
2025-06-26T05:14:38.854Z [INFO]  server-5c8dc570-440b-fc4f-6de4-8893a772e583: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c66f9aba-974a-5805-e7ca-1c97ac7934f6 Address:c66f9aba-974a-5805-e7ca-1c97ac7934f6} {Suffrage:Voter ID:server-5c8dc570-440b-fc4f-6de4-8893a772e583 Address:5c8dc570-440b-fc4f-6de4-8893a772e583}]"
2025-06-26T05:14:38.854Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:38.854Z [INFO]  server-c66f9aba-974a-5805-e7ca-1c97ac7934f6: entering follower state: follower="Node at c66f9aba-974a-5805-e7ca-1c97ac7934f6 [Follower]" leader-address= leader-id=
2025-06-26T05:14:38.854Z [INFO]  server-5c8dc570-440b-fc4f-6de4-8893a772e583: entering follower state: follower="Node at 5c8dc570-440b-fc4f-6de4-8893a772e583 [Follower]" leader-address= leader-id=
2025-06-26T05:14:38.913Z [WARN]  server-5c8dc570-440b-fc4f-6de4-8893a772e583: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:38.913Z [INFO]  server-5c8dc570-440b-fc4f-6de4-8893a772e583: entering candidate state: node="Node at 5c8dc570-440b-fc4f-6de4-8893a772e583 [Candidate]" term=2
2025-06-26T05:14:38.913Z [INFO]  server-5c8dc570-440b-fc4f-6de4-8893a772e583: election won: term=2 tally=2
2025-06-26T05:14:38.913Z [INFO]  server-5c8dc570-440b-fc4f-6de4-8893a772e583: entering leader state: leader="Node at 5c8dc570-440b-fc4f-6de4-8893a772e583 [Leader]"
2025-06-26T05:14:38.913Z [INFO]  server-5c8dc570-440b-fc4f-6de4-8893a772e583: added peer, starting replication: peer=server-c66f9aba-974a-5805-e7ca-1c97ac7934f6
2025-06-26T05:14:38.913Z [INFO]  server-5c8dc570-440b-fc4f-6de4-8893a772e583: pipelining replication: peer="{Voter server-c66f9aba-974a-5805-e7ca-1c97ac7934f6 c66f9aba-974a-5805-e7ca-1c97ac7934f6}"
2025-06-26T05:14:39.021Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:38.913938123 +0000 UTC m=+107.063048401 (1 nodes), 59.57744ms from start of poll, 59.72204ms from cluster start. Timeout at 2025-06-26 05:14:39.021624563 +0000 UTC m=+107.170734841, 107.68644ms after stability
2025-06-26T05:14:39.021Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:39.133Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:39.021831243 +0000 UTC m=+107.170941521 (1 nodes), 800ns from start of poll, 167.61516ms from cluster start. Timeout at 2025-06-26 05:14:39.133730563 +0000 UTC m=+107.282840841, 111.89932ms after stability
2025-06-26T05:14:39.133Z [INFO]  server-5c8dc570-440b-fc4f-6de4-8893a772e583: updating configuration: command=DemoteVoter server-id=server-c66f9aba-974a-5805-e7ca-1c97ac7934f6 server-addr= servers="[{Suffrage:Nonvoter ID:server-c66f9aba-974a-5805-e7ca-1c97ac7934f6 Address:c66f9aba-974a-5805-e7ca-1c97ac7934f6} {Suffrage:Voter ID:server-5c8dc570-440b-fc4f-6de4-8893a772e583 Address:5c8dc570-440b-fc4f-6de4-8893a772e583}]"
2025-06-26T05:14:39.133Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:39.241Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:39.133887763 +0000 UTC m=+107.282998041 (1 nodes), 640ns from start of poll, 279.67168ms from cluster start. Timeout at 2025-06-26 05:14:39.241572163 +0000 UTC m=+107.390682441, 107.6844ms after stability
2025-06-26T05:14:39.241Z [INFO]  server-5c8dc570-440b-fc4f-6de4-8893a772e583: aborting pipeline replication: peer="{Voter server-c66f9aba-974a-5805-e7ca-1c97ac7934f6 c66f9aba-974a-5805-e7ca-1c97ac7934f6}"
2025-06-26T05:14:39.749Z [ERROR] server-5c8dc570-440b-fc4f-6de4-8893a772e583: failed to heartbeat to: peer=c66f9aba-974a-5805-e7ca-1c97ac7934f6 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.90s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2025-06-26T05:14:39.750Z [INFO]  server-41a971c6-273f-d34d-1485-a7098723a0e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41a971c6-273f-d34d-1485-a7098723a0e3 Address:41a971c6-273f-d34d-1485-a7098723a0e3} {Suffrage:Voter ID:server-cd61b20a-0df1-99d2-eed4-ba544c4ea342 Address:cd61b20a-0df1-99d2-eed4-ba544c4ea342}]"
2025-06-26T05:14:39.750Z [INFO]  server-cd61b20a-0df1-99d2-eed4-ba544c4ea342: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41a971c6-273f-d34d-1485-a7098723a0e3 Address:41a971c6-273f-d34d-1485-a7098723a0e3} {Suffrage:Voter ID:server-cd61b20a-0df1-99d2-eed4-ba544c4ea342 Address:cd61b20a-0df1-99d2-eed4-ba544c4ea342}]"
2025-06-26T05:14:39.750Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:39.750Z [INFO]  server-41a971c6-273f-d34d-1485-a7098723a0e3: entering follower state: follower="Node at 41a971c6-273f-d34d-1485-a7098723a0e3 [Follower]" leader-address= leader-id=
2025-06-26T05:14:39.750Z [INFO]  server-cd61b20a-0df1-99d2-eed4-ba544c4ea342: entering follower state: follower="Node at cd61b20a-0df1-99d2-eed4-ba544c4ea342 [Follower]" leader-address= leader-id=
2025-06-26T05:14:39.805Z [WARN]  server-41a971c6-273f-d34d-1485-a7098723a0e3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:39.805Z [INFO]  server-41a971c6-273f-d34d-1485-a7098723a0e3: entering candidate state: node="Node at 41a971c6-273f-d34d-1485-a7098723a0e3 [Candidate]" term=2
2025-06-26T05:14:39.805Z [INFO]  server-41a971c6-273f-d34d-1485-a7098723a0e3: election won: term=2 tally=2
2025-06-26T05:14:39.805Z [INFO]  server-41a971c6-273f-d34d-1485-a7098723a0e3: entering leader state: leader="Node at 41a971c6-273f-d34d-1485-a7098723a0e3 [Leader]"
2025-06-26T05:14:39.805Z [INFO]  server-41a971c6-273f-d34d-1485-a7098723a0e3: added peer, starting replication: peer=server-cd61b20a-0df1-99d2-eed4-ba544c4ea342
2025-06-26T05:14:39.806Z [INFO]  server-41a971c6-273f-d34d-1485-a7098723a0e3: pipelining replication: peer="{Voter server-cd61b20a-0df1-99d2-eed4-ba544c4ea342 cd61b20a-0df1-99d2-eed4-ba544c4ea342}"
2025-06-26T05:14:39.921Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:39.806145843 +0000 UTC m=+107.955256121 (1 nodes), 55.91456ms from start of poll, 56.07616ms from cluster start. Timeout at 2025-06-26 05:14:39.921677403 +0000 UTC m=+108.070787681, 115.53156ms after stability
2025-06-26T05:14:39.921Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:40.030Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:39.921776723 +0000 UTC m=+108.070886961 (1 nodes), 2µs from start of poll, 171.707ms from cluster start. Timeout at 2025-06-26 05:14:40.030642803 +0000 UTC m=+108.179753081, 108.86612ms after stability
2025-06-26T05:14:40.045Z [INFO]  server-cd61b20a-0df1-99d2-eed4-ba544c4ea342: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41a971c6-273f-d34d-1485-a7098723a0e3 Address:41a971c6-273f-d34d-1485-a7098723a0e3} {Suffrage:Voter ID:server-cd61b20a-0df1-99d2-eed4-ba544c4ea342 Address:cd61b20a-0df1-99d2-eed4-ba544c4ea342}]"
2025-06-26T05:14:40.045Z [INFO]  server-41a971c6-273f-d34d-1485-a7098723a0e3: aborting pipeline replication: peer="{Voter server-cd61b20a-0df1-99d2-eed4-ba544c4ea342 cd61b20a-0df1-99d2-eed4-ba544c4ea342}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.30s)
=== RUN   TestRaft_LogStoreIsMonotonic
2025-06-26T05:14:40.046Z [INFO]  server-e4183750-c237-d3c4-c247-b634834cfa34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e4183750-c237-d3c4-c247-b634834cfa34 Address:e4183750-c237-d3c4-c247-b634834cfa34}]"
2025-06-26T05:14:40.046Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:40.046Z [INFO]  server-e4183750-c237-d3c4-c247-b634834cfa34: entering follower state: follower="Node at e4183750-c237-d3c4-c247-b634834cfa34 [Follower]" leader-address= leader-id=
2025-06-26T05:14:40.109Z [WARN]  server-e4183750-c237-d3c4-c247-b634834cfa34: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:40.109Z [INFO]  server-e4183750-c237-d3c4-c247-b634834cfa34: entering candidate state: node="Node at e4183750-c237-d3c4-c247-b634834cfa34 [Candidate]" term=2
2025-06-26T05:14:40.109Z [INFO]  server-e4183750-c237-d3c4-c247-b634834cfa34: election won: term=2 tally=1
2025-06-26T05:14:40.109Z [INFO]  server-e4183750-c237-d3c4-c247-b634834cfa34: entering leader state: leader="Node at e4183750-c237-d3c4-c247-b634834cfa34 [Leader]"
2025-06-26T05:14:40.217Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:40.109916443 +0000 UTC m=+108.259026721 (1 nodes), 63.71144ms from start of poll, 63.81472ms from cluster start. Timeout at 2025-06-26 05:14:40.217750843 +0000 UTC m=+108.366861121, 107.8344ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.17s)
=== RUN   TestRaft_CacheLogWithStoreError
2025-06-26T05:14:40.218Z [INFO]  server-86061c00-0ac1-45e7-f423-d77d20a14776: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86061c00-0ac1-45e7-f423-d77d20a14776 Address:86061c00-0ac1-45e7-f423-d77d20a14776} {Suffrage:Voter ID:server-86f2d064-ad21-5921-c9ee-e703c97f00db Address:86f2d064-ad21-5921-c9ee-e703c97f00db}]"
2025-06-26T05:14:40.218Z [INFO]  server-86f2d064-ad21-5921-c9ee-e703c97f00db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86061c00-0ac1-45e7-f423-d77d20a14776 Address:86061c00-0ac1-45e7-f423-d77d20a14776} {Suffrage:Voter ID:server-86f2d064-ad21-5921-c9ee-e703c97f00db Address:86f2d064-ad21-5921-c9ee-e703c97f00db}]"
2025-06-26T05:14:40.218Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:40.218Z [INFO]  server-86061c00-0ac1-45e7-f423-d77d20a14776: entering follower state: follower="Node at 86061c00-0ac1-45e7-f423-d77d20a14776 [Follower]" leader-address= leader-id=
2025-06-26T05:14:40.218Z [INFO]  server-86f2d064-ad21-5921-c9ee-e703c97f00db: entering follower state: follower="Node at 86f2d064-ad21-5921-c9ee-e703c97f00db [Follower]" leader-address= leader-id=
2025-06-26T05:14:40.286Z [WARN]  server-86061c00-0ac1-45e7-f423-d77d20a14776: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:40.286Z [INFO]  server-86061c00-0ac1-45e7-f423-d77d20a14776: entering candidate state: node="Node at 86061c00-0ac1-45e7-f423-d77d20a14776 [Candidate]" term=2
2025-06-26T05:14:40.286Z [INFO]  server-86061c00-0ac1-45e7-f423-d77d20a14776: election won: term=2 tally=2
2025-06-26T05:14:40.286Z [INFO]  server-86061c00-0ac1-45e7-f423-d77d20a14776: entering leader state: leader="Node at 86061c00-0ac1-45e7-f423-d77d20a14776 [Leader]"
2025-06-26T05:14:40.286Z [INFO]  server-86061c00-0ac1-45e7-f423-d77d20a14776: added peer, starting replication: peer=server-86f2d064-ad21-5921-c9ee-e703c97f00db
2025-06-26T05:14:40.286Z [INFO]  server-86061c00-0ac1-45e7-f423-d77d20a14776: pipelining replication: peer="{Voter server-86f2d064-ad21-5921-c9ee-e703c97f00db 86f2d064-ad21-5921-c9ee-e703c97f00db}"
2025-06-26T05:14:40.393Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:40.286911123 +0000 UTC m=+108.436021401 (1 nodes), 68.67472ms from start of poll, 68.8232ms from cluster start. Timeout at 2025-06-26 05:14:40.393916083 +0000 UTC m=+108.543026361, 107.00496ms after stability
2025-06-26T05:14:40.393Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:40.501Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:40.393985123 +0000 UTC m=+108.543095361 (1 nodes), 1.92µs from start of poll, 175.89716ms from cluster start. Timeout at 2025-06-26 05:14:40.501573123 +0000 UTC m=+108.650683401, 107.58804ms after stability
2025-06-26T05:14:40.502Z [ERROR] server-86f2d064-ad21-5921-c9ee-e703c97f00db: failed to append to logs: error="unable to store logs within log store, err: "some error""
2025-06-26T05:14:40.502Z [WARN]  server-86f2d064-ad21-5921-c9ee-e703c97f00db: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2025-06-26T05:14:40.502Z [INFO]  server-86061c00-0ac1-45e7-f423-d77d20a14776: aborting pipeline replication: peer="{Voter server-86f2d064-ad21-5921-c9ee-e703c97f00db 86f2d064-ad21-5921-c9ee-e703c97f00db}"
2025-06-26T05:14:40.509Z [INFO]  server-86061c00-0ac1-45e7-f423-d77d20a14776: pipelining replication: peer="{Voter server-86f2d064-ad21-5921-c9ee-e703c97f00db 86f2d064-ad21-5921-c9ee-e703c97f00db}"
2025-06-26T05:14:40.509Z [INFO]  server-86f2d064-ad21-5921-c9ee-e703c97f00db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86061c00-0ac1-45e7-f423-d77d20a14776 Address:86061c00-0ac1-45e7-f423-d77d20a14776} {Suffrage:Voter ID:server-86f2d064-ad21-5921-c9ee-e703c97f00db Address:86f2d064-ad21-5921-c9ee-e703c97f00db}]"
2025-06-26T05:14:40.509Z [INFO]  server-86061c00-0ac1-45e7-f423-d77d20a14776: aborting pipeline replication: peer="{Voter server-86f2d064-ad21-5921-c9ee-e703c97f00db 86f2d064-ad21-5921-c9ee-e703c97f00db}"
--- PASS: TestRaft_CacheLogWithStoreError (0.29s)
=== RUN   TestRaft_ReloadConfig
2025-06-26T05:14:40.510Z [INFO]  server-91d02810-6b1c-5f1c-9126-8c29683b4fba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91d02810-6b1c-5f1c-9126-8c29683b4fba Address:91d02810-6b1c-5f1c-9126-8c29683b4fba}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2025-06-26T05:14:40.510Z [INFO]  server-60b3a638-06af-997f-cfc0-f7f5f1671d4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60b3a638-06af-997f-cfc0-f7f5f1671d4e Address:60b3a638-06af-997f-cfc0-f7f5f1671d4e}]"
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2025-06-26T05:14:40.510Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2025-06-26T05:14:40.510Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2 Address:57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2} {Suffrage:Voter ID:server-c0a7cb82-acc9-ceec-a735-d16798b86d27 Address:c0a7cb82-acc9-ceec-a735-d16798b86d27} {Suffrage:Voter ID:server-fd802229-068d-707b-0a93-de661caf0f2b Address:fd802229-068d-707b-0a93-de661caf0f2b}]"
2025-06-26T05:14:40.510Z [INFO]  server-c0a7cb82-acc9-ceec-a735-d16798b86d27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2 Address:57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2} {Suffrage:Voter ID:server-c0a7cb82-acc9-ceec-a735-d16798b86d27 Address:c0a7cb82-acc9-ceec-a735-d16798b86d27} {Suffrage:Voter ID:server-fd802229-068d-707b-0a93-de661caf0f2b Address:fd802229-068d-707b-0a93-de661caf0f2b}]"
2025-06-26T05:14:40.510Z [INFO]  server-fd802229-068d-707b-0a93-de661caf0f2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2 Address:57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2} {Suffrage:Voter ID:server-c0a7cb82-acc9-ceec-a735-d16798b86d27 Address:c0a7cb82-acc9-ceec-a735-d16798b86d27} {Suffrage:Voter ID:server-fd802229-068d-707b-0a93-de661caf0f2b Address:fd802229-068d-707b-0a93-de661caf0f2b}]"
2025-06-26T05:14:40.510Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:40.510Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: entering follower state: follower="Node at 57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2 [Follower]" leader-address= leader-id=
2025-06-26T05:14:40.510Z [INFO]  server-c0a7cb82-acc9-ceec-a735-d16798b86d27: entering follower state: follower="Node at c0a7cb82-acc9-ceec-a735-d16798b86d27 [Follower]" leader-address= leader-id=
2025-06-26T05:14:40.510Z [INFO]  server-fd802229-068d-707b-0a93-de661caf0f2b: entering follower state: follower="Node at fd802229-068d-707b-0a93-de661caf0f2b [Follower]" leader-address= leader-id=
2025-06-26T05:14:40.573Z [WARN]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:40.573Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: entering candidate state: node="Node at 57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2 [Candidate]" term=2
2025-06-26T05:14:40.573Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: election won: term=2 tally=2
2025-06-26T05:14:40.573Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: entering leader state: leader="Node at 57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2 [Leader]"
2025-06-26T05:14:40.573Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: added peer, starting replication: peer=server-c0a7cb82-acc9-ceec-a735-d16798b86d27
2025-06-26T05:14:40.573Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: added peer, starting replication: peer=server-fd802229-068d-707b-0a93-de661caf0f2b
2025-06-26T05:14:40.573Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: pipelining replication: peer="{Voter server-c0a7cb82-acc9-ceec-a735-d16798b86d27 c0a7cb82-acc9-ceec-a735-d16798b86d27}"
2025-06-26T05:14:40.573Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: pipelining replication: peer="{Voter server-fd802229-068d-707b-0a93-de661caf0f2b fd802229-068d-707b-0a93-de661caf0f2b}"
2025-06-26T05:14:40.685Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:40.573785243 +0000 UTC m=+108.722895521 (1 nodes), 62.9658ms from start of poll, 63.13476ms from cluster start. Timeout at 2025-06-26 05:14:40.685573323 +0000 UTC m=+108.834683601, 111.78808ms after stability
2025-06-26T05:14:40.690Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:40.797Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:40.690926763 +0000 UTC m=+108.840037041 (2 nodes), 2.08µs from start of poll, 180.27628ms from cluster start. Timeout at 2025-06-26 05:14:40.797591323 +0000 UTC m=+108.946701601, 106.66456ms after stability
2025-06-26T05:14:40.797Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: updating configuration: command=RemoveServer server-id=server-c0a7cb82-acc9-ceec-a735-d16798b86d27 server-addr= servers="[{Suffrage:Voter ID:server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2 Address:57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2} {Suffrage:Voter ID:server-fd802229-068d-707b-0a93-de661caf0f2b Address:fd802229-068d-707b-0a93-de661caf0f2b}]"
2025-06-26T05:14:40.797Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: removed peer, stopping replication: peer=server-c0a7cb82-acc9-ceec-a735-d16798b86d27 last-index=3
2025-06-26T05:14:40.797Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: aborting pipeline replication: peer="{Voter server-c0a7cb82-acc9-ceec-a735-d16798b86d27 c0a7cb82-acc9-ceec-a735-d16798b86d27}"
2025-06-26T05:14:40.909Z [ERROR] server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: failed to pipeline appendEntries: peer="{Voter server-fd802229-068d-707b-0a93-de661caf0f2b fd802229-068d-707b-0a93-de661caf0f2b}" error="append pipeline closed"
2025-06-26T05:14:40.909Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: aborting pipeline replication: peer="{Voter server-fd802229-068d-707b-0a93-de661caf0f2b fd802229-068d-707b-0a93-de661caf0f2b}"
2025-06-26T05:14:40.917Z [ERROR] server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: failed to heartbeat to: peer=fd802229-068d-707b-0a93-de661caf0f2b backoff time=10ms error="failed to connect to peer: fd802229-068d-707b-0a93-de661caf0f2b"
2025-06-26T05:14:40.921Z [WARN]  server-c0a7cb82-acc9-ceec-a735-d16798b86d27: not part of stable configuration, aborting election
2025-06-26T05:14:40.921Z [ERROR] server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: failed to appendEntries to: peer="{Voter server-fd802229-068d-707b-0a93-de661caf0f2b fd802229-068d-707b-0a93-de661caf0f2b}" error="failed to connect to peer: fd802229-068d-707b-0a93-de661caf0f2b"
2025-06-26T05:14:40.941Z [ERROR] server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: failed to heartbeat to: peer=fd802229-068d-707b-0a93-de661caf0f2b backoff time=10ms error="failed to connect to peer: fd802229-068d-707b-0a93-de661caf0f2b"
2025-06-26T05:14:40.941Z [ERROR] server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: failed to appendEntries to: peer="{Voter server-fd802229-068d-707b-0a93-de661caf0f2b fd802229-068d-707b-0a93-de661caf0f2b}" error="failed to connect to peer: fd802229-068d-707b-0a93-de661caf0f2b"
2025-06-26T05:14:40.953Z [WARN]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: failed to contact: server-id=server-fd802229-068d-707b-0a93-de661caf0f2b time=52.1282ms
2025-06-26T05:14:40.953Z [WARN]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: failed to contact quorum of nodes, stepping down
2025-06-26T05:14:40.953Z [INFO]  server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: entering follower state: follower="Node at 57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2 [Follower]" leader-address= leader-id=
2025-06-26T05:14:40.969Z [ERROR] server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: failed to heartbeat to: peer=fd802229-068d-707b-0a93-de661caf0f2b backoff time=10ms error="failed to connect to peer: fd802229-068d-707b-0a93-de661caf0f2b"
2025-06-26T05:14:40.969Z [ERROR] server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2: failed to appendEntries to: peer="{Voter server-fd802229-068d-707b-0a93-de661caf0f2b fd802229-068d-707b-0a93-de661caf0f2b}" error="failed to connect to peer: fd802229-068d-707b-0a93-de661caf0f2b"
2025-06-26T05:14:40.981Z [WARN]  server-fd802229-068d-707b-0a93-de661caf0f2b: heartbeat timeout reached, starting election: last-leader-addr=57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2 last-leader-id=server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2
2025-06-26T05:14:40.981Z [INFO]  server-fd802229-068d-707b-0a93-de661caf0f2b: entering candidate state: node="Node at fd802229-068d-707b-0a93-de661caf0f2b [Candidate]" term=3
2025-06-26T05:14:40.981Z [ERROR] server-fd802229-068d-707b-0a93-de661caf0f2b: failed to make requestVote RPC: target="{Voter server-57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2 57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2}" error="failed to connect to peer: 57a6b65a-7a7c-1d76-4b18-f5c2b752e8a2" term=3
2025-06-26T05:14:41.013Z [WARN]  server-fd802229-068d-707b-0a93-de661caf0f2b: rejecting vote request since node is not in configuration: from=c0a7cb82-acc9-ceec-a735-d16798b86d27
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.50s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2025-06-26T05:14:41.014Z [INFO]  server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 Address:dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36} {Suffrage:Voter ID:server-c455f9f7-1e13-7f73-a9d1-334e8f754024 Address:c455f9f7-1e13-7f73-a9d1-334e8f754024} {Suffrage:Voter ID:server-7d6fa2e2-4824-216c-e775-3f1c7315a562 Address:7d6fa2e2-4824-216c-e775-3f1c7315a562}]"
2025-06-26T05:14:41.014Z [INFO]  server-c455f9f7-1e13-7f73-a9d1-334e8f754024: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 Address:dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36} {Suffrage:Voter ID:server-c455f9f7-1e13-7f73-a9d1-334e8f754024 Address:c455f9f7-1e13-7f73-a9d1-334e8f754024} {Suffrage:Voter ID:server-7d6fa2e2-4824-216c-e775-3f1c7315a562 Address:7d6fa2e2-4824-216c-e775-3f1c7315a562}]"
2025-06-26T05:14:41.014Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 Address:dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36} {Suffrage:Voter ID:server-c455f9f7-1e13-7f73-a9d1-334e8f754024 Address:c455f9f7-1e13-7f73-a9d1-334e8f754024} {Suffrage:Voter ID:server-7d6fa2e2-4824-216c-e775-3f1c7315a562 Address:7d6fa2e2-4824-216c-e775-3f1c7315a562}]"
2025-06-26T05:14:41.014Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:41.014Z [INFO]  server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36: entering follower state: follower="Node at dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 [Follower]" leader-address= leader-id=
2025-06-26T05:14:41.014Z [INFO]  server-c455f9f7-1e13-7f73-a9d1-334e8f754024: entering follower state: follower="Node at c455f9f7-1e13-7f73-a9d1-334e8f754024 [Follower]" leader-address= leader-id=
2025-06-26T05:14:41.014Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: entering follower state: follower="Node at 7d6fa2e2-4824-216c-e775-3f1c7315a562 [Follower]" leader-address= leader-id=
2025-06-26T05:14:41.077Z [WARN]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:41.078Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: entering candidate state: node="Node at 7d6fa2e2-4824-216c-e775-3f1c7315a562 [Candidate]" term=2
2025-06-26T05:14:41.078Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: election won: term=2 tally=2
2025-06-26T05:14:41.078Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: entering leader state: leader="Node at 7d6fa2e2-4824-216c-e775-3f1c7315a562 [Leader]"
2025-06-26T05:14:41.078Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: added peer, starting replication: peer=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36
2025-06-26T05:14:41.078Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: added peer, starting replication: peer=server-c455f9f7-1e13-7f73-a9d1-334e8f754024
2025-06-26T05:14:41.078Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: pipelining replication: peer="{Voter server-c455f9f7-1e13-7f73-a9d1-334e8f754024 c455f9f7-1e13-7f73-a9d1-334e8f754024}"
2025-06-26T05:14:41.078Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: pipelining replication: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}"
2025-06-26T05:14:41.183Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:41.078245323 +0000 UTC m=+109.227355601 (1 nodes), 63.76484ms from start of poll, 63.96884ms from cluster start. Timeout at 2025-06-26 05:14:41.183279243 +0000 UTC m=+109.332389521, 105.03392ms after stability
2025-06-26T05:14:41.189Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:41.297Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:41.190004563 +0000 UTC m=+109.339114841 (2 nodes), 2.08µs from start of poll, 175.72808ms from cluster start. Timeout at 2025-06-26 05:14:41.297911803 +0000 UTC m=+109.447022081, 107.90724ms after stability
2025-06-26T05:14:41.298Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to pipeline appendEntries: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}" error="append pipeline closed"
2025-06-26T05:14:41.298Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: aborting pipeline replication: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}"
2025-06-26T05:14:41.301Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to heartbeat to: peer=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 backoff time=10ms error="failed to connect to peer: dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36"
2025-06-26T05:14:41.305Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to appendEntries to: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}" error="failed to connect to peer: dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36"
2025-06-26T05:14:41.325Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to heartbeat to: peer=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 backoff time=10ms error="failed to connect to peer: dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36"
2025-06-26T05:14:41.325Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to appendEntries to: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}" error="failed to connect to peer: dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36"
2025-06-26T05:14:41.345Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to heartbeat to: peer=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 backoff time=10ms error="failed to connect to peer: dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36"
2025-06-26T05:14:41.346Z [WARN]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to contact: server-id=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 time=52.09264ms
2025-06-26T05:14:41.349Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to appendEntries to: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}" error="failed to connect to peer: dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36"
2025-06-26T05:14:41.366Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to heartbeat to: peer=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 backoff time=20ms error="failed to connect to peer: dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36"
2025-06-26T05:14:41.385Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to appendEntries to: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}" error="failed to connect to peer: dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36"
2025-06-26T05:14:41.393Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to heartbeat to: peer=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 backoff time=25ms error="failed to connect to peer: dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36"
2025-06-26T05:14:41.397Z [WARN]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to contact: server-id=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 time=103.93776ms
2025-06-26T05:14:41.405Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: updating configuration: command=RemoveServer server-id=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 server-addr= servers="[{Suffrage:Voter ID:server-c455f9f7-1e13-7f73-a9d1-334e8f754024 Address:c455f9f7-1e13-7f73-a9d1-334e8f754024} {Suffrage:Voter ID:server-7d6fa2e2-4824-216c-e775-3f1c7315a562 Address:7d6fa2e2-4824-216c-e775-3f1c7315a562}]"
2025-06-26T05:14:41.405Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: removed peer, stopping replication: peer=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 last-index=3
2025-06-26T05:14:41.406Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: updating configuration: command=AddNonvoter server-id=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 server-addr=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 servers="[{Suffrage:Voter ID:server-c455f9f7-1e13-7f73-a9d1-334e8f754024 Address:c455f9f7-1e13-7f73-a9d1-334e8f754024} {Suffrage:Voter ID:server-7d6fa2e2-4824-216c-e775-3f1c7315a562 Address:7d6fa2e2-4824-216c-e775-3f1c7315a562} {Suffrage:Nonvoter ID:server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 Address:dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}]"
2025-06-26T05:14:41.406Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: added peer, starting replication: peer=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36
2025-06-26T05:14:41.406Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to appendEntries to: peer="{Nonvoter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}" error="failed to connect to peer: dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36"
2025-06-26T05:14:41.414Z [WARN]  server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36: heartbeat timeout reached, starting election: last-leader-addr=7d6fa2e2-4824-216c-e775-3f1c7315a562 last-leader-id=server-7d6fa2e2-4824-216c-e775-3f1c7315a562
2025-06-26T05:14:41.414Z [INFO]  server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36: entering candidate state: node="Node at dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 [Candidate]" term=23
2025-06-26T05:14:41.414Z [WARN]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: rejecting vote request since we have a leader: from=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 leader=7d6fa2e2-4824-216c-e775-3f1c7315a562 leader-id=server-7d6fa2e2-4824-216c-e775-3f1c7315a562
2025-06-26T05:14:41.414Z [WARN]  server-c455f9f7-1e13-7f73-a9d1-334e8f754024: rejecting vote request since we have a leader: from=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 leader=7d6fa2e2-4824-216c-e775-3f1c7315a562 leader-id=server-7d6fa2e2-4824-216c-e775-3f1c7315a562
2025-06-26T05:14:41.429Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: peer has newer term, stopping replication: peer="{Nonvoter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}"
2025-06-26T05:14:41.429Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: entering follower state: follower="Node at 7d6fa2e2-4824-216c-e775-3f1c7315a562 [Follower]" leader-address= leader-id=
2025-06-26T05:14:41.429Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: aborting pipeline replication: peer="{Voter server-c455f9f7-1e13-7f73-a9d1-334e8f754024 c455f9f7-1e13-7f73-a9d1-334e8f754024}"
2025-06-26T05:14:41.437Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: peer has newer term, stopping replication: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}"
2025-06-26T05:14:41.473Z [WARN]  server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36: Election timeout reached, restarting election
2025-06-26T05:14:41.473Z [INFO]  server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36: entering candidate state: node="Node at dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 [Candidate]" term=24
2025-06-26T05:14:41.473Z [WARN]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: rejecting vote request since node is not a voter: from=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36
2025-06-26T05:14:41.473Z [WARN]  server-c455f9f7-1e13-7f73-a9d1-334e8f754024: rejecting vote request since we have a leader: from=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 leader=7d6fa2e2-4824-216c-e775-3f1c7315a562 leader-id=server-7d6fa2e2-4824-216c-e775-3f1c7315a562
2025-06-26T05:14:41.489Z [WARN]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:41.489Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: entering candidate state: node="Node at 7d6fa2e2-4824-216c-e775-3f1c7315a562 [Candidate]" term=25
2025-06-26T05:14:41.489Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: election won: term=25 tally=2
2025-06-26T05:14:41.489Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: entering leader state: leader="Node at 7d6fa2e2-4824-216c-e775-3f1c7315a562 [Leader]"
2025-06-26T05:14:41.489Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: added peer, starting replication: peer=server-c455f9f7-1e13-7f73-a9d1-334e8f754024
2025-06-26T05:14:41.489Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: added peer, starting replication: peer=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36
2025-06-26T05:14:41.489Z [WARN]  server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2025-06-26T05:14:41.489Z [INFO]  server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36: entering follower state: follower="Node at dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 [Follower]" leader-address=7d6fa2e2-4824-216c-e775-3f1c7315a562 leader-id=server-7d6fa2e2-4824-216c-e775-3f1c7315a562
2025-06-26T05:14:41.489Z [WARN]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: appendEntries rejected, sending older logs: peer="{Nonvoter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}" next=3
2025-06-26T05:14:41.490Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: pipelining replication: peer="{Nonvoter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}"
2025-06-26T05:14:41.490Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: pipelining replication: peer="{Voter server-c455f9f7-1e13-7f73-a9d1-334e8f754024 c455f9f7-1e13-7f73-a9d1-334e8f754024}"
2025-06-26T05:14:41.514Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:41.621Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:41.514309883 +0000 UTC m=+109.663420161 (1 nodes), 32.6µs from start of poll, 500.0334ms from cluster start. Timeout at 2025-06-26 05:14:41.621615523 +0000 UTC m=+109.770725801, 107.30564ms after stability
2025-06-26T05:14:41.673Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: updating configuration: command=RemoveServer server-id=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 server-addr= servers="[{Suffrage:Voter ID:server-c455f9f7-1e13-7f73-a9d1-334e8f754024 Address:c455f9f7-1e13-7f73-a9d1-334e8f754024} {Suffrage:Voter ID:server-7d6fa2e2-4824-216c-e775-3f1c7315a562 Address:7d6fa2e2-4824-216c-e775-3f1c7315a562}]"
2025-06-26T05:14:41.673Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: removed peer, stopping replication: peer=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 last-index=106
2025-06-26T05:14:41.673Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: aborting pipeline replication: peer="{Nonvoter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}"
2025-06-26T05:14:41.674Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: updating configuration: command=AddVoter server-id=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 server-addr=dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 servers="[{Suffrage:Voter ID:server-c455f9f7-1e13-7f73-a9d1-334e8f754024 Address:c455f9f7-1e13-7f73-a9d1-334e8f754024} {Suffrage:Voter ID:server-7d6fa2e2-4824-216c-e775-3f1c7315a562 Address:7d6fa2e2-4824-216c-e775-3f1c7315a562} {Suffrage:Voter ID:server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 Address:dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}]"
2025-06-26T05:14:41.674Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: added peer, starting replication: peer=server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36
2025-06-26T05:14:41.674Z [WARN]  server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2025-06-26T05:14:41.674Z [WARN]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: appendEntries rejected, sending older logs: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}" next=107
2025-06-26T05:14:41.674Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: pipelining replication: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}"
2025-06-26T05:14:42.749Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:42.857Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:42.749744083 +0000 UTC m=+110.898854361 (1 nodes), 2.4µs from start of poll, 1.7354676s from cluster start. Timeout at 2025-06-26 05:14:42.857615803 +0000 UTC m=+111.006726081, 107.87172ms after stability
2025-06-26T05:14:42.857Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: aborting pipeline replication: peer="{Voter server-dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36 dbf05caf-a7f4-7c73-dd80-9e4cae2e2b36}"
2025-06-26T05:14:42.857Z [INFO]  server-7d6fa2e2-4824-216c-e775-3f1c7315a562: aborting pipeline replication: peer="{Voter server-c455f9f7-1e13-7f73-a9d1-334e8f754024 c455f9f7-1e13-7f73-a9d1-334e8f754024}"
2025-06-26T05:14:43.357Z [ERROR] server-7d6fa2e2-4824-216c-e775-3f1c7315a562: failed to heartbeat to: peer=c455f9f7-1e13-7f73-a9d1-334e8f754024 backoff time=10ms error="command timed out"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (2.34s)
=== RUN   TestRaft_FollowerRemovalNoElection
2025-06-26T05:14:43.358Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-892e8141-ef45-8efd-e5c5-bf025aa2012e Address:892e8141-ef45-8efd-e5c5-bf025aa2012e} {Suffrage:Voter ID:server-a7211e0e-e0b8-a570-4b39-ae634eec7082 Address:a7211e0e-e0b8-a570-4b39-ae634eec7082} {Suffrage:Voter ID:server-51aacef0-9086-5272-e7fd-3941156871f7 Address:51aacef0-9086-5272-e7fd-3941156871f7}]"
2025-06-26T05:14:43.358Z [INFO]  server-a7211e0e-e0b8-a570-4b39-ae634eec7082: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-892e8141-ef45-8efd-e5c5-bf025aa2012e Address:892e8141-ef45-8efd-e5c5-bf025aa2012e} {Suffrage:Voter ID:server-a7211e0e-e0b8-a570-4b39-ae634eec7082 Address:a7211e0e-e0b8-a570-4b39-ae634eec7082} {Suffrage:Voter ID:server-51aacef0-9086-5272-e7fd-3941156871f7 Address:51aacef0-9086-5272-e7fd-3941156871f7}]"
2025-06-26T05:14:43.358Z [INFO]  server-51aacef0-9086-5272-e7fd-3941156871f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-892e8141-ef45-8efd-e5c5-bf025aa2012e Address:892e8141-ef45-8efd-e5c5-bf025aa2012e} {Suffrage:Voter ID:server-a7211e0e-e0b8-a570-4b39-ae634eec7082 Address:a7211e0e-e0b8-a570-4b39-ae634eec7082} {Suffrage:Voter ID:server-51aacef0-9086-5272-e7fd-3941156871f7 Address:51aacef0-9086-5272-e7fd-3941156871f7}]"
2025-06-26T05:14:43.358Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:43.358Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: entering follower state: follower="Node at 892e8141-ef45-8efd-e5c5-bf025aa2012e [Follower]" leader-address= leader-id=
2025-06-26T05:14:43.358Z [INFO]  server-a7211e0e-e0b8-a570-4b39-ae634eec7082: entering follower state: follower="Node at a7211e0e-e0b8-a570-4b39-ae634eec7082 [Follower]" leader-address= leader-id=
2025-06-26T05:14:43.358Z [INFO]  server-51aacef0-9086-5272-e7fd-3941156871f7: entering follower state: follower="Node at 51aacef0-9086-5272-e7fd-3941156871f7 [Follower]" leader-address= leader-id=
2025-06-26T05:14:43.469Z [WARN]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:43.469Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: entering candidate state: node="Node at 892e8141-ef45-8efd-e5c5-bf025aa2012e [Candidate]" term=2
2025-06-26T05:14:43.469Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: election won: term=2 tally=2
2025-06-26T05:14:43.469Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: entering leader state: leader="Node at 892e8141-ef45-8efd-e5c5-bf025aa2012e [Leader]"
2025-06-26T05:14:43.469Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: added peer, starting replication: peer=server-a7211e0e-e0b8-a570-4b39-ae634eec7082
2025-06-26T05:14:43.469Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: added peer, starting replication: peer=server-51aacef0-9086-5272-e7fd-3941156871f7
2025-06-26T05:14:43.469Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: pipelining replication: peer="{Voter server-51aacef0-9086-5272-e7fd-3941156871f7 51aacef0-9086-5272-e7fd-3941156871f7}"
2025-06-26T05:14:43.469Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: pipelining replication: peer="{Voter server-a7211e0e-e0b8-a570-4b39-ae634eec7082 a7211e0e-e0b8-a570-4b39-ae634eec7082}"
2025-06-26T05:14:43.678Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:43.469891203 +0000 UTC m=+111.619001481 (1 nodes), 111.5362ms from start of poll, 111.74016ms from cluster start. Timeout at 2025-06-26 05:14:43.678062363 +0000 UTC m=+111.827172601, 208.17112ms after stability
2025-06-26T05:14:43.678Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:43.885Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:43.678123243 +0000 UTC m=+111.827233521 (1 nodes), 1.68µs from start of poll, 319.9722ms from cluster start. Timeout at 2025-06-26 05:14:43.885637483 +0000 UTC m=+112.034747761, 207.51424ms after stability
2025-06-26T05:14:43.893Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:44.105Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:43.893727003 +0000 UTC m=+112.042837241 (2 nodes), 2.08µs from start of poll, 535.57592ms from cluster start. Timeout at 2025-06-26 05:14:44.105604963 +0000 UTC m=+112.254715241, 211.878ms after stability
2025-06-26T05:14:44.105Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:44.313Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:44.105729323 +0000 UTC m=+112.254839601 (1 nodes), 640ns from start of poll, 747.57828ms from cluster start. Timeout at 2025-06-26 05:14:44.313819883 +0000 UTC m=+112.462930161, 208.09056ms after stability
    raft_test.go:2961: [INFO] restarting Node at a7211e0e-e0b8-a570-4b39-ae634eec7082 [Follower]
2025-06-26T05:14:44.314Z [INFO]  server-a7211e0e-e0b8-a570-4b39-ae634eec7082: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-892e8141-ef45-8efd-e5c5-bf025aa2012e Address:892e8141-ef45-8efd-e5c5-bf025aa2012e} {Suffrage:Voter ID:server-a7211e0e-e0b8-a570-4b39-ae634eec7082 Address:a7211e0e-e0b8-a570-4b39-ae634eec7082} {Suffrage:Voter ID:server-51aacef0-9086-5272-e7fd-3941156871f7 Address:51aacef0-9086-5272-e7fd-3941156871f7}]"
2025-06-26T05:14:44.314Z [INFO]  server-a7211e0e-e0b8-a570-4b39-ae634eec7082: entering follower state: follower="Node at a7211e0e-e0b8-a570-4b39-ae634eec7082 [Follower]" leader-address= leader-id=
2025-06-26T05:14:44.566Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: aborting pipeline replication: peer="{Voter server-51aacef0-9086-5272-e7fd-3941156871f7 51aacef0-9086-5272-e7fd-3941156871f7}"
2025-06-26T05:14:44.998Z [ERROR] server-892e8141-ef45-8efd-e5c5-bf025aa2012e: failed to pipeline appendEntries: peer="{Voter server-a7211e0e-e0b8-a570-4b39-ae634eec7082 a7211e0e-e0b8-a570-4b39-ae634eec7082}" error="command enqueue timeout"
2025-06-26T05:14:44.998Z [INFO]  server-892e8141-ef45-8efd-e5c5-bf025aa2012e: aborting pipeline replication: peer="{Voter server-a7211e0e-e0b8-a570-4b39-ae634eec7082 a7211e0e-e0b8-a570-4b39-ae634eec7082}"
2025-06-26T05:14:45.078Z [ERROR] server-892e8141-ef45-8efd-e5c5-bf025aa2012e: failed to heartbeat to: peer=a7211e0e-e0b8-a570-4b39-ae634eec7082 backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.72s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2025-06-26T05:14:45.078Z [INFO]  server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 Address:7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1} {Suffrage:Voter ID:server-13a091e7-1c6e-1484-68da-adc34d4041af Address:13a091e7-1c6e-1484-68da-adc34d4041af} {Suffrage:Voter ID:server-f94c076d-1213-42a4-6abc-b4d99cd12df5 Address:f94c076d-1213-42a4-6abc-b4d99cd12df5}]"
2025-06-26T05:14:45.078Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 Address:7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1} {Suffrage:Voter ID:server-13a091e7-1c6e-1484-68da-adc34d4041af Address:13a091e7-1c6e-1484-68da-adc34d4041af} {Suffrage:Voter ID:server-f94c076d-1213-42a4-6abc-b4d99cd12df5 Address:f94c076d-1213-42a4-6abc-b4d99cd12df5}]"
2025-06-26T05:14:45.078Z [INFO]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 Address:7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1} {Suffrage:Voter ID:server-13a091e7-1c6e-1484-68da-adc34d4041af Address:13a091e7-1c6e-1484-68da-adc34d4041af} {Suffrage:Voter ID:server-f94c076d-1213-42a4-6abc-b4d99cd12df5 Address:f94c076d-1213-42a4-6abc-b4d99cd12df5}]"
2025-06-26T05:14:45.078Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:45.078Z [INFO]  server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1: entering follower state: follower="Node at 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 [Follower]" leader-address= leader-id=
2025-06-26T05:14:45.078Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering follower state: follower="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Follower]" leader-address= leader-id=
2025-06-26T05:14:45.078Z [INFO]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: entering follower state: follower="Node at f94c076d-1213-42a4-6abc-b4d99cd12df5 [Follower]" leader-address= leader-id=
2025-06-26T05:14:45.133Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:45.133Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=2
2025-06-26T05:14:45.133Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: election won: term=2 tally=2
2025-06-26T05:14:45.133Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering leader state: leader="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Leader]"
2025-06-26T05:14:45.133Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: added peer, starting replication: peer=server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1
2025-06-26T05:14:45.133Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: added peer, starting replication: peer=server-f94c076d-1213-42a4-6abc-b4d99cd12df5
2025-06-26T05:14:45.133Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: pipelining replication: peer="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}"
2025-06-26T05:14:45.133Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: pipelining replication: peer="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}"
2025-06-26T05:14:45.134Z [WARN]  server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1: rejecting vote request since our last term is greater: candidate=13a091e7-1c6e-1484-68da-adc34d4041af last-term=2 last-candidate-term=1
2025-06-26T05:14:45.242Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:45.134072963 +0000 UTC m=+113.283183241 (1 nodes), 55.17816ms from start of poll, 55.38268ms from cluster start. Timeout at 2025-06-26 05:14:45.242168843 +0000 UTC m=+113.391279361, 108.09612ms after stability
2025-06-26T05:14:45.242Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-06-26T05:14:45.349Z [INFO]  cluster: stable state for Leader reached at 2025-06-26 05:14:45.242243083 +0000 UTC m=+113.391353361 (1 nodes), 1.56µs from start of poll, 163.5528ms from cluster start. Timeout at 2025-06-26 05:14:45.349617443 +0000 UTC m=+113.498727721, 107.37436ms after stability
2025-06-26T05:14:45.357Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-06-26T05:14:45.469Z [INFO]  cluster: stable state for Follower reached at 2025-06-26 05:14:45.357952483 +0000 UTC m=+113.507062761 (2 nodes), 1.28µs from start of poll, 279.2622ms from cluster start. Timeout at 2025-06-26 05:14:45.469940003 +0000 UTC m=+113.619050281, 111.98752ms after stability
2025-06-26T05:14:45.486Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to heartbeat to: peer=f94c076d-1213-42a4-6abc-b4d99cd12df5 backoff time=10ms error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5"
2025-06-26T05:14:45.486Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to pipeline appendEntries: peer="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="append pipeline closed"
2025-06-26T05:14:45.486Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: aborting pipeline replication: peer="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}"
2025-06-26T05:14:45.486Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to pipeline appendEntries: peer="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="append pipeline closed"
2025-06-26T05:14:45.486Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: aborting pipeline replication: peer="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}"
2025-06-26T05:14:45.486Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to heartbeat to: peer=7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 backoff time=10ms error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1"
2025-06-26T05:14:45.493Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to appendEntries to: peer="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5"
2025-06-26T05:14:45.497Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to appendEntries to: peer="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1"
2025-06-26T05:14:45.505Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to heartbeat to: peer=f94c076d-1213-42a4-6abc-b4d99cd12df5 backoff time=10ms error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5"
2025-06-26T05:14:45.505Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to heartbeat to: peer=7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 backoff time=10ms error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1"
2025-06-26T05:14:45.513Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to appendEntries to: peer="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5"
2025-06-26T05:14:45.521Z [WARN]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: heartbeat timeout reached, starting election: last-leader-addr=13a091e7-1c6e-1484-68da-adc34d4041af last-leader-id=server-13a091e7-1c6e-1484-68da-adc34d4041af
2025-06-26T05:14:45.521Z [INFO]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: entering candidate state: node="Node at f94c076d-1213-42a4-6abc-b4d99cd12df5 [Candidate]" term=3
2025-06-26T05:14:45.521Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to make requestVote RPC: target="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af" term=3
2025-06-26T05:14:45.521Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to appendEntries to: peer="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1"
2025-06-26T05:14:45.521Z [WARN]  server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1: rejecting vote request since we have a leader: from=f94c076d-1213-42a4-6abc-b4d99cd12df5 leader=13a091e7-1c6e-1484-68da-adc34d4041af leader-id=server-13a091e7-1c6e-1484-68da-adc34d4041af
2025-06-26T05:14:45.521Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to contact: server-id=server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 time=51.94408ms
2025-06-26T05:14:45.521Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to contact: server-id=server-f94c076d-1213-42a4-6abc-b4d99cd12df5 time=51.94952ms
2025-06-26T05:14:45.521Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to contact quorum of nodes, stepping down
2025-06-26T05:14:45.521Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering follower state: follower="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Follower]" leader-address= leader-id=
2025-06-26T05:14:45.529Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to heartbeat to: peer=f94c076d-1213-42a4-6abc-b4d99cd12df5 backoff time=10ms error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5"
2025-06-26T05:14:45.533Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to appendEntries to: peer="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5"
2025-06-26T05:14:45.585Z [WARN]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: Election timeout reached, restarting election
2025-06-26T05:14:45.585Z [INFO]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: entering candidate state: node="Node at f94c076d-1213-42a4-6abc-b4d99cd12df5 [Candidate]" term=4
2025-06-26T05:14:45.585Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to make requestVote RPC: target="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af" term=4
2025-06-26T05:14:45.586Z [WARN]  server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1: heartbeat timeout reached, starting election: last-leader-addr=13a091e7-1c6e-1484-68da-adc34d4041af last-leader-id=server-13a091e7-1c6e-1484-68da-adc34d4041af
2025-06-26T05:14:45.586Z [INFO]  server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1: entering candidate state: node="Node at 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 [Candidate]" term=3
2025-06-26T05:14:45.586Z [INFO]  server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1: entering follower state: follower="Node at 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 [Follower]" leader-address= leader-id=
2025-06-26T05:14:45.586Z [INFO]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: election won: term=4 tally=2
2025-06-26T05:14:45.586Z [INFO]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: entering leader state: leader="Node at f94c076d-1213-42a4-6abc-b4d99cd12df5 [Leader]"
2025-06-26T05:14:45.586Z [INFO]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: added peer, starting replication: peer=server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1
2025-06-26T05:14:45.586Z [INFO]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: added peer, starting replication: peer=server-13a091e7-1c6e-1484-68da-adc34d4041af
2025-06-26T05:14:45.586Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.586Z [ERROR] server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1: failed to make requestVote RPC: target="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af" term=3
2025-06-26T05:14:45.586Z [INFO]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: pipelining replication: peer="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}"
2025-06-26T05:14:45.593Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=10ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.609Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.617Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-06-26T05:14:45.617Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=3
2025-06-26T05:14:45.617Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=3
2025-06-26T05:14:45.617Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=3
2025-06-26T05:14:45.618Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=10ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.629Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.641Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=10ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.645Z [WARN]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to contact: server-id=server-13a091e7-1c6e-1484-68da-adc34d4041af time=59.6866ms
2025-06-26T05:14:45.661Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.665Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=20ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.693Z [WARN]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to contact: server-id=server-13a091e7-1c6e-1484-68da-adc34d4041af time=107.5282ms
2025-06-26T05:14:45.693Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.717Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:45.717Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=4
2025-06-26T05:14:45.717Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=4
2025-06-26T05:14:45.717Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=4
2025-06-26T05:14:45.717Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.729Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.769Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.801Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:45.801Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=5
2025-06-26T05:14:45.801Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=5
2025-06-26T05:14:45.801Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=5
2025-06-26T05:14:45.805Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.817Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.849Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.881Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:45.881Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=6
2025-06-26T05:14:45.881Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=6
2025-06-26T05:14:45.881Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=6
2025-06-26T05:14:45.897Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.933Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:45.957Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:45.957Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=7
2025-06-26T05:14:45.957Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=7
2025-06-26T05:14:45.957Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=7
2025-06-26T05:14:45.974Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.002Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.013Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.053Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.053Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=8
2025-06-26T05:14:46.053Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=8
2025-06-26T05:14:46.053Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=8
2025-06-26T05:14:46.057Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.093Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.113Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.113Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=9
2025-06-26T05:14:46.113Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=9
2025-06-26T05:14:46.113Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=9
2025-06-26T05:14:46.145Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.181Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.181Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=10
2025-06-26T05:14:46.181Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=10
2025-06-26T05:14:46.181Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=10
2025-06-26T05:14:46.181Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.221Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.265Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.265Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=11
2025-06-26T05:14:46.265Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=11
2025-06-26T05:14:46.266Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.266Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=11
2025-06-26T05:14:46.301Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.325Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.325Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=12
2025-06-26T05:14:46.329Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=12
2025-06-26T05:14:46.329Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=12
2025-06-26T05:14:46.337Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.337Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.393Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.429Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.429Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=13
2025-06-26T05:14:46.430Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=13
2025-06-26T05:14:46.430Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=13
2025-06-26T05:14:46.433Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.469Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.491Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.491Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=14
2025-06-26T05:14:46.491Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=14
2025-06-26T05:14:46.491Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=14
2025-06-26T05:14:46.504Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.537Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.551Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.551Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=15
2025-06-26T05:14:46.551Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=15
2025-06-26T05:14:46.551Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=15
2025-06-26T05:14:46.573Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.613Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.649Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.653Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.653Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=16
2025-06-26T05:14:46.653Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=16
2025-06-26T05:14:46.653Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=16
2025-06-26T05:14:46.697Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.713Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.713Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=17
2025-06-26T05:14:46.713Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=17
2025-06-26T05:14:46.713Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=17
2025-06-26T05:14:46.741Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.775Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.775Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=18
2025-06-26T05:14:46.775Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=18
2025-06-26T05:14:46.775Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=18
2025-06-26T05:14:46.781Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.817Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.857Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.873Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.873Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=19
2025-06-26T05:14:46.873Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=19
2025-06-26T05:14:46.873Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=19
2025-06-26T05:14:46.897Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.937Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.973Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:46.973Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:46.973Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=20
2025-06-26T05:14:46.973Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=20
2025-06-26T05:14:46.973Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=20
2025-06-26T05:14:46.985Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.057Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.071Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.071Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=21
2025-06-26T05:14:47.071Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=21
2025-06-26T05:14:47.071Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=21
2025-06-26T05:14:47.097Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.129Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.129Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=22
2025-06-26T05:14:47.129Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=22
2025-06-26T05:14:47.129Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=22
2025-06-26T05:14:47.138Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.177Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.185Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.185Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=23
2025-06-26T05:14:47.185Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=23
2025-06-26T05:14:47.185Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=23
2025-06-26T05:14:47.213Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.249Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.281Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.281Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=24
2025-06-26T05:14:47.281Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=24
2025-06-26T05:14:47.281Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=24
2025-06-26T05:14:47.293Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.329Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.337Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.337Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=25
2025-06-26T05:14:47.337Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=25
2025-06-26T05:14:47.337Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=25
2025-06-26T05:14:47.369Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.393Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.393Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=26
2025-06-26T05:14:47.394Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=26
2025-06-26T05:14:47.394Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=26
2025-06-26T05:14:47.417Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.457Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.481Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.481Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=27
2025-06-26T05:14:47.481Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=27
2025-06-26T05:14:47.481Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=27
2025-06-26T05:14:47.501Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.541Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.549Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.549Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=28
2025-06-26T05:14:47.549Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=28
2025-06-26T05:14:47.549Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=28
2025-06-26T05:14:47.581Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.621Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.625Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.625Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=29
2025-06-26T05:14:47.625Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=29
2025-06-26T05:14:47.625Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=29
2025-06-26T05:14:47.657Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.697Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.714Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=30
2025-06-26T05:14:47.714Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=30
2025-06-26T05:14:47.714Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=30
2025-06-26T05:14:47.725Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.761Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.789Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.789Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=31
2025-06-26T05:14:47.789Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=31
2025-06-26T05:14:47.789Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=31
2025-06-26T05:14:47.797Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.837Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.848Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.848Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=32
2025-06-26T05:14:47.848Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=32
2025-06-26T05:14:47.848Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=32
2025-06-26T05:14:47.885Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.921Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:47.921Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=33
2025-06-26T05:14:47.921Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=33
2025-06-26T05:14:47.921Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:47.921Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=33
2025-06-26T05:14:47.970Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.010Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.031Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.050Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=34
2025-06-26T05:14:48.050Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=34
2025-06-26T05:14:48.050Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=34
2025-06-26T05:14:48.062Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.097Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.101Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.101Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=35
2025-06-26T05:14:48.101Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=35
2025-06-26T05:14:48.101Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=35
2025-06-26T05:14:48.133Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.173Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.193Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.193Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=36
2025-06-26T05:14:48.193Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=36
2025-06-26T05:14:48.193Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=36
2025-06-26T05:14:48.217Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.261Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.270Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.270Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=37
2025-06-26T05:14:48.270Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=37
2025-06-26T05:14:48.270Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=37
2025-06-26T05:14:48.289Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.298Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.322Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.322Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=38
2025-06-26T05:14:48.322Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=38
2025-06-26T05:14:48.322Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=38
2025-06-26T05:14:48.337Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.385Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.413Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.413Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=39
2025-06-26T05:14:48.413Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=39
2025-06-26T05:14:48.413Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=39
2025-06-26T05:14:48.421Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.457Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.497Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.499Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.499Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=40
2025-06-26T05:14:48.499Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=40
2025-06-26T05:14:48.499Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=40
2025-06-26T05:14:48.537Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.585Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.601Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.601Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=41
2025-06-26T05:14:48.601Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=41
2025-06-26T05:14:48.601Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=41
2025-06-26T05:14:48.625Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.665Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.701Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.701Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=42
2025-06-26T05:14:48.701Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=42
2025-06-26T05:14:48.701Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=42
2025-06-26T05:14:48.709Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.749Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.773Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.773Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=43
2025-06-26T05:14:48.773Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=43
2025-06-26T05:14:48.773Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=43
2025-06-26T05:14:48.789Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.825Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.865Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.881Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.881Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=44
2025-06-26T05:14:48.881Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=44
2025-06-26T05:14:48.881Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=44
2025-06-26T05:14:48.901Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.933Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:48.933Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=45
2025-06-26T05:14:48.933Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=45
2025-06-26T05:14:48.933Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=45
2025-06-26T05:14:48.941Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:48.982Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.018Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.034Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=46
2025-06-26T05:14:49.034Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=46
2025-06-26T05:14:49.034Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=46
2025-06-26T05:14:49.041Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.081Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.121Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.125Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.125Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=47
2025-06-26T05:14:49.125Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=47
2025-06-26T05:14:49.125Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=47
2025-06-26T05:14:49.157Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.201Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.225Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.225Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=48
2025-06-26T05:14:49.225Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=48
2025-06-26T05:14:49.225Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=48
2025-06-26T05:14:49.241Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.277Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.309Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.309Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=49
2025-06-26T05:14:49.309Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=49
2025-06-26T05:14:49.309Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=49
2025-06-26T05:14:49.313Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.349Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.385Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.401Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.401Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=50
2025-06-26T05:14:49.401Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=50
2025-06-26T05:14:49.401Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=50
2025-06-26T05:14:49.441Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.457Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.457Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=51
2025-06-26T05:14:49.457Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=51
2025-06-26T05:14:49.458Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=51
2025-06-26T05:14:49.481Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.521Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.553Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.553Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=52
2025-06-26T05:14:49.553Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=52
2025-06-26T05:14:49.553Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=52
2025-06-26T05:14:49.557Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.597Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.637Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.653Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.653Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=53
2025-06-26T05:14:49.653Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=53
2025-06-26T05:14:49.653Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=53
2025-06-26T05:14:49.681Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.725Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.733Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.733Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=54
2025-06-26T05:14:49.733Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=54
2025-06-26T05:14:49.733Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=54
2025-06-26T05:14:49.761Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.793Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.793Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=55
2025-06-26T05:14:49.793Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=55
2025-06-26T05:14:49.793Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=55
2025-06-26T05:14:49.801Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.841Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.877Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.889Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.889Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=56
2025-06-26T05:14:49.889Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=56
2025-06-26T05:14:49.889Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=56
2025-06-26T05:14:49.917Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.957Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:49.985Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:49.985Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=57
2025-06-26T05:14:49.985Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=57
2025-06-26T05:14:49.985Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=57
2025-06-26T05:14:49.993Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.043Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.066Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:50.066Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=58
2025-06-26T05:14:50.066Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=58
2025-06-26T05:14:50.066Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=58
2025-06-26T05:14:50.082Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.121Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:50.121Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=59
2025-06-26T05:14:50.121Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=59
2025-06-26T05:14:50.121Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=59
2025-06-26T05:14:50.129Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.165Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.185Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:50.185Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=60
2025-06-26T05:14:50.185Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=60
2025-06-26T05:14:50.185Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=60
2025-06-26T05:14:50.205Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.258Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.266Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:50.266Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=61
2025-06-26T05:14:50.266Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=61
2025-06-26T05:14:50.266Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=61
2025-06-26T05:14:50.293Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.333Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.333Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:50.333Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=62
2025-06-26T05:14:50.333Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=62
2025-06-26T05:14:50.334Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=62
2025-06-26T05:14:50.365Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.409Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:50.409Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=63
2025-06-26T05:14:50.409Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=63
2025-06-26T05:14:50.409Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=63
2025-06-26T05:14:50.409Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.445Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.485Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.501Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:50.501Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=64
2025-06-26T05:14:50.501Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=64
2025-06-26T05:14:50.501Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=64
2025-06-26T05:14:50.525Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.569Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to heartbeat to: peer=13a091e7-1c6e-1484-68da-adc34d4041af backoff time=25ms error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
2025-06-26T05:14:50.579Z [WARN]  server-13a091e7-1c6e-1484-68da-adc34d4041af: Election timeout reached, restarting election
2025-06-26T05:14:50.579Z [INFO]  server-13a091e7-1c6e-1484-68da-adc34d4041af: entering candidate state: node="Node at 13a091e7-1c6e-1484-68da-adc34d4041af [Candidate]" term=65
2025-06-26T05:14:50.579Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-f94c076d-1213-42a4-6abc-b4d99cd12df5 f94c076d-1213-42a4-6abc-b4d99cd12df5}" error="failed to connect to peer: f94c076d-1213-42a4-6abc-b4d99cd12df5" term=65
2025-06-26T05:14:50.579Z [ERROR] server-13a091e7-1c6e-1484-68da-adc34d4041af: failed to make requestVote RPC: target="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}" error="failed to connect to peer: 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1" term=65
    raft_test.go:3028: 
        	Error Trace:	/build/reproducible-path/golang-github-hashicorp-raft-1.5.0/obj-aarch64-linux-gnu/src/github.com/hashicorp/raft/raft_test.go:3028
        	Error:      	Not equal: 
        	            	expected: 0x1
        	            	actual  : 0x0
        	Test:       	TestRaft_VoteWithNoIDNoAddr
2025-06-26T05:14:50.609Z [INFO]  server-f94c076d-1213-42a4-6abc-b4d99cd12df5: aborting pipeline replication: peer="{Voter server-7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1 7cc2c757-da80-1c60-f0e2-8cc8e74fe9a1}"
2025-06-26T05:14:50.610Z [ERROR] server-f94c076d-1213-42a4-6abc-b4d99cd12df5: failed to appendEntries to: peer="{Voter server-13a091e7-1c6e-1484-68da-adc34d4041af 13a091e7-1c6e-1484-68da-adc34d4041af}" error="failed to connect to peer: 13a091e7-1c6e-1484-68da-adc34d4041af"
--- FAIL: TestRaft_VoteWithNoIDNoAddr (5.53s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2025-06-26T05:14:50.610Z [INFO]  127.0.0.1:41743 :: starting node: addr=127.0.0.1:41743
2025-06-26T05:14:50.610Z [INFO]  127.0.0.1:41743 :: initial configuration: index=0 servers=[]
2025-06-26T05:14:50.610Z [INFO]  127.0.0.1:41743 :: entering follower state: follower="Node at 127.0.0.1:41743 [Follower]" leader-address= leader-id=
2025-06-26T05:14:50.701Z [WARN]  127.0.0.1:41743 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2025-06-26T05:14:50.761Z [INFO]  127.0.0.1:44323 :: starting node: addr=127.0.0.1:44323
2025-06-26T05:14:50.761Z [INFO]  127.0.0.1:44323 :: initial configuration: index=0 servers=[]
2025-06-26T05:14:50.761Z [INFO]  127.0.0.1:44323 :: entering follower state: follower="Node at 127.0.0.1:44323 [Follower]" leader-address= leader-id=
2025-06-26T05:14:50.841Z [WARN]  127.0.0.1:44323 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2025-06-26T05:14:50.914Z [INFO]  127.0.0.1:39135 :: starting node: addr=127.0.0.1:39135
2025-06-26T05:14:50.914Z [INFO]  127.0.0.1:39135 :: initial configuration: index=0 servers=[]
2025-06-26T05:14:50.914Z [INFO]  127.0.0.1:39135 :: entering follower state: follower="Node at 127.0.0.1:39135 [Follower]" leader-address= leader-id=
2025-06-26T05:14:50.974Z [WARN]  127.0.0.1:39135 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.46s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2025-06-26T05:14:51.067Z [INFO]  127.0.0.1:36707 :: starting node: addr=127.0.0.1:36707
2025-06-26T05:14:51.067Z [INFO]  127.0.0.1:36707 :: initial configuration: index=0 servers=[]
2025-06-26T05:14:51.067Z [INFO]  127.0.0.1:36707 :: entering follower state: follower="Node at 127.0.0.1:36707 [Follower]" leader-address= leader-id=
2025-06-26T05:14:51.067Z [WARN]  127.0.0.1:36707 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
FAIL
FAIL	github.com/hashicorp/raft	119.421s
FAIL
dh_auto_test: error: cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1
make[1]: *** [debian/rules:17: override_dh_auto_test] Error 25
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
make: *** [debian/rules:14: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
I: copying local configuration
E: Failed autobuilding of package
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/1601072 and its subdirectories
Thu May 23 22:51:55 UTC 2024  W: No second build log, what happened?