Mon Jan 8 10:28:00 UTC 2024 I: starting to build golang-github-hashicorp-raft/bookworm/amd64 on jenkins on '2024-01-08 10:27' Mon Jan 8 10:28:00 UTC 2024 I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/amd64_2/12510/console.log Mon Jan 8 10:28:00 UTC 2024 I: Downloading source for bookworm/golang-github-hashicorp-raft=1.1.2-1 --2024-01-08 10:28:01-- http://cdn-fastly.deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.1.2-1.dsc Connecting to 78.137.99.97:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 2408 (2.4K) [text/prs.lines.tag] Saving to: ‘golang-github-hashicorp-raft_1.1.2-1.dsc’ 0K .. 100% 249M=0s 2024-01-08 10:28:01 (249 MB/s) - ‘golang-github-hashicorp-raft_1.1.2-1.dsc’ saved [2408/2408] Mon Jan 8 10:28:01 UTC 2024 I: golang-github-hashicorp-raft_1.1.2-1.dsc -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Format: 3.0 (quilt) Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.1.2-1 Maintainer: Debian Go Packaging Team Uploaders: Tim Potter , Dmitry Smirnov , Homepage: https://github.com/hashicorp/raft Standards-Version: 4.4.1 Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Testsuite: autopkgtest-pkg-go Build-Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) Package-List: golang-github-hashicorp-raft-dev deb devel optional arch=all Checksums-Sha1: 6eb88916d5b8568cc5e49dfd778e53f425f02612 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 3efa08a30605d234327a7c1293b36ecde63ffbbe 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Checksums-Sha256: 6c3383ee430aa6b421cc4cb4cbc485b0e612bda2e94e88d416d13d65d375dc89 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 3877bf052d5719266221189613502262d1be1af152c736d3dc8ed49734ad6dd8 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Files: 65844d5b08afa2d62ea4dedaf43e6047 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 036528358012213daa0588e86be2a3d4 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Go-Import-Path: github.com/hashicorp/raft -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEULx8+TnSDCcqawZWUra72VOWjRsFAl4kBCwACgkQUra72VOW jRsPQBAAxzZChwaTUFMUIQz9GgdwyR5o2SjXw48JJ//lfygslStaiz8ju52lp33x a1d56obRwTy2QmY2s6+oV3Mq4sQWs7U/1PGnywKeA71Us10A4AAv5d9wMRq1eqzp PcyVdhfM0zFcmJf8Wr93Mi8wbCr9PT+oTxc59+zWDruS7xCoe3aDtaekVfb8QZyj OZPsaTtneSxKjqk+V5Jyx8X6Y6P2cwv89GDd4nxTxapw8H38e2cBk7BsYmXUNXye 6N7dOChDN+x6gqgBU0O97oex+QOpd4LNoFBl2gEi7kukw+jrjrdyKQCANPh0CKx0 K4MPq2BeI3qKJV21l2z7S30BSjduvyktC7efckw5CX4zgT9/kJ5e7ZAf/og+M2vc CoCV+aoAH1OOWbkPu0dsuJiA5z1ZkKGa8ZB5MgvUentk7LVd/HaXW9QDdRTNZT8z Jdh6/wnpSn8ZZEhAixTtqNt64s0sNWmh4krF0WgGNZ1djzYd6gVI1LqlJTaSRKFo nxGsEe+jnC6kv2u8XpLG+xevTKcu4UIFHq2ebpanFvxzqxAv1dWTBDDI13NApMZb Po5huhcdLapUW4NGW/3pkdJkpcY5u26zH6YvTxDICU/tHXDo5/BqI9E1jEj+5m5J XrNqpKLFjW2I9gRcmT5vy3O4edLCPBFs6MGQBK7u1V+nEkh8pJc= =IObb -----END PGP SIGNATURE----- Mon Jan 8 10:28:01 UTC 2024 I: Checking whether the package is not for us Mon Jan 8 10:28:01 UTC 2024 I: Starting 1st build on remote node ionos5-amd64.debian.net. Mon Jan 8 10:28:01 UTC 2024 I: Preparing to do remote build '1' on ionos5-amd64.debian.net. Mon Jan 8 10:29:24 UTC 2024 I: Deleting $TMPDIR on ionos5-amd64.debian.net. I: pbuilder: network access will be disabled during build I: Current time: Sun Feb 9 04:51:11 -12 2025 I: pbuilder-time-stamp: 1739119871 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bookworm-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.1.2-1.dsc] I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz] I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: Signature made Sun Jan 19 07:24:28 2020 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.1.2-1.dsc: no acceptable signature found dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/2584380/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='amd64' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=16 ' DISTRIBUTION='bookworm' HOME='/root' HOST_ARCH='amd64' IFS=' ' INVOCATION_ID='928459a88b304a5da08cf117b482941b' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='2584380' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.5KHihXX9/pbuilderrc_ZUv2 --distribution bookworm --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bookworm-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.5KHihXX9/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' SUDO_GID='110' SUDO_UID='105' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://85.184.249.68:3128' I: uname -a Linux ionos5-amd64 6.5.0-0.deb12.4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.10-1~bpo12+1 (2023-11-23) x86_64 GNU/Linux I: ls -l /bin total 5632 -rwxr-xr-x 1 root root 1265648 Apr 23 2023 bash -rwxr-xr-x 3 root root 39224 Sep 19 2022 bunzip2 -rwxr-xr-x 3 root root 39224 Sep 19 2022 bzcat lrwxrwxrwx 1 root root 6 Sep 19 2022 bzcmp -> bzdiff -rwxr-xr-x 1 root root 2225 Sep 19 2022 bzdiff lrwxrwxrwx 1 root root 6 Sep 19 2022 bzegrep -> bzgrep -rwxr-xr-x 1 root root 4893 Nov 27 2021 bzexe lrwxrwxrwx 1 root root 6 Sep 19 2022 bzfgrep -> bzgrep -rwxr-xr-x 1 root root 3775 Sep 19 2022 bzgrep -rwxr-xr-x 3 root root 39224 Sep 19 2022 bzip2 -rwxr-xr-x 1 root root 14568 Sep 19 2022 bzip2recover lrwxrwxrwx 1 root root 6 Sep 19 2022 bzless -> bzmore -rwxr-xr-x 1 root root 1297 Sep 19 2022 bzmore -rwxr-xr-x 1 root root 44016 Sep 20 2022 cat -rwxr-xr-x 1 root root 68656 Sep 20 2022 chgrp -rwxr-xr-x 1 root root 64496 Sep 20 2022 chmod -rwxr-xr-x 1 root root 72752 Sep 20 2022 chown -rwxr-xr-x 1 root root 151152 Sep 20 2022 cp -rwxr-xr-x 1 root root 125640 Jan 5 2023 dash -rwxr-xr-x 1 root root 121904 Sep 20 2022 date -rwxr-xr-x 1 root root 89240 Sep 20 2022 dd -rwxr-xr-x 1 root root 102200 Sep 20 2022 df -rwxr-xr-x 1 root root 151344 Sep 20 2022 dir -rwxr-xr-x 1 root root 88656 Mar 23 2023 dmesg lrwxrwxrwx 1 root root 8 Dec 19 2022 dnsdomainname -> hostname lrwxrwxrwx 1 root root 8 Dec 19 2022 domainname -> hostname -rwxr-xr-x 1 root root 43856 Sep 20 2022 echo -rwxr-xr-x 1 root root 41 Jan 24 2023 egrep -rwxr-xr-x 1 root root 35664 Sep 20 2022 false -rwxr-xr-x 1 root root 41 Jan 24 2023 fgrep -rwxr-xr-x 1 root root 85600 Mar 23 2023 findmnt -rwsr-xr-x 1 root root 35128 Mar 23 2023 fusermount -rwxr-xr-x 1 root root 203152 Jan 24 2023 grep -rwxr-xr-x 2 root root 2346 Apr 10 2022 gunzip -rwxr-xr-x 1 root root 6447 Apr 10 2022 gzexe -rwxr-xr-x 1 root root 98136 Apr 10 2022 gzip -rwxr-xr-x 1 root root 22680 Dec 19 2022 hostname -rwxr-xr-x 1 root root 72824 Sep 20 2022 ln -rwxr-xr-x 1 root root 53024 Mar 23 2023 login -rwxr-xr-x 1 root root 151344 Sep 20 2022 ls -rwxr-xr-x 1 root root 207168 Mar 23 2023 lsblk -rwxr-xr-x 1 root root 97552 Sep 20 2022 mkdir -rwxr-xr-x 1 root root 72912 Sep 20 2022 mknod -rwxr-xr-x 1 root root 43952 Sep 20 2022 mktemp -rwxr-xr-x 1 root root 59712 Mar 23 2023 more -rwsr-xr-x 1 root root 59704 Mar 23 2023 mount -rwxr-xr-x 1 root root 18744 Mar 23 2023 mountpoint -rwxr-xr-x 1 root root 142968 Sep 20 2022 mv lrwxrwxrwx 1 root root 8 Dec 19 2022 nisdomainname -> hostname lrwxrwxrwx 1 root root 14 Apr 3 2023 pidof -> /sbin/killall5 -rwxr-xr-x 1 root root 43952 Sep 20 2022 pwd lrwxrwxrwx 1 root root 4 Apr 23 2023 rbash -> bash -rwxr-xr-x 1 root root 52112 Sep 20 2022 readlink -rwxr-xr-x 1 root root 72752 Sep 20 2022 rm -rwxr-xr-x 1 root root 56240 Sep 20 2022 rmdir -rwxr-xr-x 1 root root 27560 Jul 28 2023 run-parts -rwxr-xr-x 1 root root 126424 Jan 5 2023 sed lrwxrwxrwx 1 root root 4 Jan 5 2023 sh -> dash -rwxr-xr-x 1 root root 43888 Sep 20 2022 sleep -rwxr-xr-x 1 root root 85008 Sep 20 2022 stty -rwsr-xr-x 1 root root 72000 Mar 23 2023 su -rwxr-xr-x 1 root root 39824 Sep 20 2022 sync -rwxr-xr-x 1 root root 531984 Apr 6 2023 tar -rwxr-xr-x 1 root root 14520 Jul 28 2023 tempfile -rwxr-xr-x 1 root root 109616 Sep 20 2022 touch -rwxr-xr-x 1 root root 35664 Sep 20 2022 true -rwxr-xr-x 1 root root 14568 Mar 23 2023 ulockmgr_server -rwsr-xr-x 1 root root 35128 Mar 23 2023 umount -rwxr-xr-x 1 root root 43888 Sep 20 2022 uname -rwxr-xr-x 2 root root 2346 Apr 10 2022 uncompress -rwxr-xr-x 1 root root 151344 Sep 20 2022 vdir -rwxr-xr-x 1 root root 72024 Mar 23 2023 wdctl lrwxrwxrwx 1 root root 8 Dec 19 2022 ypdomainname -> hostname -rwxr-xr-x 1 root root 1984 Apr 10 2022 zcat -rwxr-xr-x 1 root root 1678 Apr 10 2022 zcmp -rwxr-xr-x 1 root root 6460 Apr 10 2022 zdiff -rwxr-xr-x 1 root root 29 Apr 10 2022 zegrep -rwxr-xr-x 1 root root 29 Apr 10 2022 zfgrep -rwxr-xr-x 1 root root 2081 Apr 10 2022 zforce -rwxr-xr-x 1 root root 8103 Apr 10 2022 zgrep -rwxr-xr-x 1 root root 2206 Apr 10 2022 zless -rwxr-xr-x 1 root root 1842 Apr 10 2022 zmore -rwxr-xr-x 1 root root 4577 Apr 10 2022 znew I: user script /srv/workspace/pbuilder/2584380/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: amd64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 18148 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper (>= 12~); however: Package debhelper is not installed. pbuilder-satisfydepends-dummy depends on dh-golang; however: Package dh-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.19-go{a} golang-1.19-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-3-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-diff-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-go{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-text-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-protobuf-extensions-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 98 newly installed, 0 to remove and 0 not upgraded. Need to get 118 MB of archives. After unpacking 785 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian bookworm/main amd64 sensible-utils all 0.0.17+nmu1 [19.0 kB] Get: 2 http://deb.debian.org/debian bookworm/main amd64 libmagic-mgc amd64 1:5.44-3 [305 kB] Get: 3 http://deb.debian.org/debian bookworm/main amd64 libmagic1 amd64 1:5.44-3 [104 kB] Get: 4 http://deb.debian.org/debian bookworm/main amd64 file amd64 1:5.44-3 [42.5 kB] Get: 5 http://deb.debian.org/debian bookworm/main amd64 gettext-base amd64 0.21-12 [160 kB] Get: 6 http://deb.debian.org/debian bookworm/main amd64 libuchardet0 amd64 0.0.7-1 [67.8 kB] Get: 7 http://deb.debian.org/debian bookworm/main amd64 groff-base amd64 1.22.4-10 [916 kB] Get: 8 http://deb.debian.org/debian bookworm/main amd64 bsdextrautils amd64 2.38.1-5+b1 [86.6 kB] Get: 9 http://deb.debian.org/debian bookworm/main amd64 libpipeline1 amd64 1.5.7-1 [38.5 kB] Get: 10 http://deb.debian.org/debian bookworm/main amd64 man-db amd64 2.11.2-2 [1386 kB] Get: 11 http://deb.debian.org/debian bookworm/main amd64 m4 amd64 1.4.19-3 [287 kB] Get: 12 http://deb.debian.org/debian bookworm/main amd64 autoconf all 2.71-3 [332 kB] Get: 13 http://deb.debian.org/debian bookworm/main amd64 autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian bookworm/main amd64 automake all 1:1.16.5-1.3 [823 kB] Get: 15 http://deb.debian.org/debian bookworm/main amd64 autopoint all 0.21-12 [495 kB] Get: 16 http://deb.debian.org/debian bookworm/main amd64 libdebhelper-perl all 13.11.4 [81.2 kB] Get: 17 http://deb.debian.org/debian bookworm/main amd64 libtool all 2.4.7-5 [517 kB] Get: 18 http://deb.debian.org/debian bookworm/main amd64 dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian bookworm/main amd64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian bookworm/main amd64 libsub-override-perl all 0.09-4 [9304 B] Get: 21 http://deb.debian.org/debian bookworm/main amd64 libfile-stripnondeterminism-perl all 1.13.1-1 [19.4 kB] Get: 22 http://deb.debian.org/debian bookworm/main amd64 dh-strip-nondeterminism all 1.13.1-1 [8620 B] Get: 23 http://deb.debian.org/debian bookworm/main amd64 libelf1 amd64 0.188-2.1 [174 kB] Get: 24 http://deb.debian.org/debian bookworm/main amd64 dwz amd64 0.15-1 [109 kB] Get: 25 http://deb.debian.org/debian bookworm/main amd64 libicu72 amd64 72.1-3 [9376 kB] Get: 26 http://deb.debian.org/debian bookworm/main amd64 libxml2 amd64 2.9.14+dfsg-1.3~deb12u1 [687 kB] Get: 27 http://deb.debian.org/debian bookworm/main amd64 gettext amd64 0.21-12 [1300 kB] Get: 28 http://deb.debian.org/debian bookworm/main amd64 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 29 http://deb.debian.org/debian bookworm/main amd64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 30 http://deb.debian.org/debian bookworm/main amd64 debhelper all 13.11.4 [942 kB] Get: 31 http://deb.debian.org/debian bookworm/main amd64 dh-golang all 1.59 [27.8 kB] Get: 32 http://deb.debian.org/debian bookworm/main amd64 golang-1.19-src all 1.19.8-2 [18.3 MB] Get: 33 http://deb.debian.org/debian bookworm/main amd64 golang-1.19-go amd64 1.19.8-2 [62.7 MB] Get: 34 http://deb.debian.org/debian bookworm/main amd64 golang-src all 2:1.19~1 [5756 B] Get: 35 http://deb.debian.org/debian bookworm/main amd64 golang-go amd64 2:1.19~1 [45.0 kB] Get: 36 http://deb.debian.org/debian bookworm/main amd64 golang-any amd64 2:1.19~1 [5876 B] Get: 37 http://deb.debian.org/debian bookworm/main amd64 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B] Get: 38 http://deb.debian.org/debian bookworm/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 39 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 40 http://deb.debian.org/debian bookworm/main amd64 golang-golang-x-sys-dev all 0.3.0-1 [393 kB] Get: 41 http://deb.debian.org/debian bookworm/main amd64 golang-github-mattn-go-isatty-dev all 0.0.17-1 [6016 B] Get: 42 http://deb.debian.org/debian bookworm/main amd64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 43 http://deb.debian.org/debian bookworm/main amd64 golang-github-fatih-color-dev all 1.7.0-1 [11.4 kB] Get: 44 http://deb.debian.org/debian bookworm/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 45 http://deb.debian.org/debian bookworm/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 46 http://deb.debian.org/debian bookworm/main amd64 golang-github-stretchr-objx-dev all 0.5.0-1 [25.5 kB] Get: 47 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 48 http://deb.debian.org/debian bookworm/main amd64 golang-github-stretchr-testify-dev all 1.8.1-1 [67.9 kB] Get: 49 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 50 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 51 http://deb.debian.org/debian bookworm/main amd64 golang-github-pkg-errors-dev all 0.9.1-2 [13.2 kB] Get: 52 http://deb.debian.org/debian bookworm/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 53 http://deb.debian.org/debian bookworm/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 54 http://deb.debian.org/debian bookworm/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 55 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.2-2 [8480 B] Get: 56 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 57 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 58 http://deb.debian.org/debian bookworm/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 59 http://deb.debian.org/debian bookworm/main amd64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 60 http://deb.debian.org/debian bookworm/main amd64 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B] Get: 61 http://deb.debian.org/debian bookworm/main amd64 golang-github-golang-protobuf-1-3-dev all 1.3.5-4 [234 kB] Get: 62 http://deb.debian.org/debian bookworm/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 63 http://deb.debian.org/debian bookworm/main amd64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 64 http://deb.debian.org/debian bookworm/main amd64 golang-github-json-iterator-go-dev all 1.1.12-1 [62.2 kB] Get: 65 http://deb.debian.org/debian bookworm/main amd64 golang-github-prometheus-client-model-dev all 0.3.0-3 [10.5 kB] Get: 66 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB] Get: 67 http://deb.debian.org/debian bookworm/main amd64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 68 http://deb.debian.org/debian bookworm/main amd64 golang-golang-x-text-dev all 0.7.0-1 [3863 kB] Get: 69 http://deb.debian.org/debian bookworm/main amd64 golang-golang-x-net-dev all 1:0.7.0+dfsg-1 [731 kB] Get: 70 http://deb.debian.org/debian bookworm/main amd64 golang-github-aws-aws-sdk-go-dev all 1.44.133-1 [10.5 MB] Get: 71 http://deb.debian.org/debian bookworm/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 72 http://deb.debian.org/debian bookworm/main amd64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 73 http://deb.debian.org/debian bookworm/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 74 http://deb.debian.org/debian bookworm/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 75 http://deb.debian.org/debian bookworm/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB] Get: 76 http://deb.debian.org/debian bookworm/main amd64 golang-golang-x-oauth2-dev all 0.3.0-1 [35.0 kB] Get: 77 http://deb.debian.org/debian bookworm/main amd64 golang-google-protobuf-dev all 1.28.1-3 [542 kB] Get: 78 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 79 http://deb.debian.org/debian bookworm/main amd64 golang-protobuf-extensions-dev all 1.0.1-3 [29.5 kB] Get: 80 http://deb.debian.org/debian bookworm/main amd64 golang-github-prometheus-common-dev all 0.39.0-2 [110 kB] Get: 81 http://deb.debian.org/debian bookworm/main amd64 golang-golang-x-sync-dev all 0.1.0-1 [19.7 kB] Get: 82 http://deb.debian.org/debian bookworm/main amd64 golang-github-prometheus-procfs-dev all 0.8.0-3 [148 kB] Get: 83 http://deb.debian.org/debian bookworm/main amd64 golang-github-prometheus-client-golang-dev all 1.14.0-3 [174 kB] Get: 84 http://deb.debian.org/debian bookworm/main amd64 golang-github-armon-go-metrics-dev all 0.3.4-1 [30.5 kB] Get: 85 http://deb.debian.org/debian bookworm/main amd64 golang-github-creack-pty-dev all 1.1.18-1 [9796 B] Get: 86 http://deb.debian.org/debian bookworm/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 87 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 88 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 89 http://deb.debian.org/debian bookworm/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg-10 [20.3 kB] Get: 90 http://deb.debian.org/debian bookworm/main amd64 libsasl2-2 amd64 2.1.28+dfsg-10 [59.7 kB] Get: 91 http://deb.debian.org/debian bookworm/main amd64 libsasl2-dev amd64 2.1.28+dfsg-10 [235 kB] Get: 92 http://deb.debian.org/debian bookworm/main amd64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 93 http://deb.debian.org/debian bookworm/main amd64 golang-github-pkg-diff-dev all 0.0~git20210226.20ebb0f-1 [93.6 kB] Get: 94 http://deb.debian.org/debian bookworm/main amd64 golang-github-rogpeppe-go-internal-dev all 1.9.0-1 [117 kB] Get: 95 http://deb.debian.org/debian bookworm/main amd64 golang-github-kr-pretty-dev all 0.3.0-1 [14.9 kB] Get: 96 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 97 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 98 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 118 MB in 2s (61.0 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 18148 files and directories currently installed.) Preparing to unpack .../00-sensible-utils_0.0.17+nmu1_all.deb ... Unpacking sensible-utils (0.0.17+nmu1) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../01-libmagic-mgc_1%3a5.44-3_amd64.deb ... Unpacking libmagic-mgc (1:5.44-3) ... Selecting previously unselected package libmagic1:amd64. Preparing to unpack .../02-libmagic1_1%3a5.44-3_amd64.deb ... Unpacking libmagic1:amd64 (1:5.44-3) ... Selecting previously unselected package file. Preparing to unpack .../03-file_1%3a5.44-3_amd64.deb ... Unpacking file (1:5.44-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../04-gettext-base_0.21-12_amd64.deb ... Unpacking gettext-base (0.21-12) ... Selecting previously unselected package libuchardet0:amd64. Preparing to unpack .../05-libuchardet0_0.0.7-1_amd64.deb ... Unpacking libuchardet0:amd64 (0.0.7-1) ... Selecting previously unselected package groff-base. Preparing to unpack .../06-groff-base_1.22.4-10_amd64.deb ... Unpacking groff-base (1.22.4-10) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../07-bsdextrautils_2.38.1-5+b1_amd64.deb ... Unpacking bsdextrautils (2.38.1-5+b1) ... Selecting previously unselected package libpipeline1:amd64. Preparing to unpack .../08-libpipeline1_1.5.7-1_amd64.deb ... Unpacking libpipeline1:amd64 (1.5.7-1) ... Selecting previously unselected package man-db. Preparing to unpack .../09-man-db_2.11.2-2_amd64.deb ... Unpacking man-db (2.11.2-2) ... Selecting previously unselected package m4. Preparing to unpack .../10-m4_1.4.19-3_amd64.deb ... Unpacking m4 (1.4.19-3) ... Selecting previously unselected package autoconf. Preparing to unpack .../11-autoconf_2.71-3_all.deb ... Unpacking autoconf (2.71-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../12-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../13-automake_1%3a1.16.5-1.3_all.deb ... Unpacking automake (1:1.16.5-1.3) ... Selecting previously unselected package autopoint. Preparing to unpack .../14-autopoint_0.21-12_all.deb ... Unpacking autopoint (0.21-12) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../15-libdebhelper-perl_13.11.4_all.deb ... Unpacking libdebhelper-perl (13.11.4) ... Selecting previously unselected package libtool. Preparing to unpack .../16-libtool_2.4.7-5_all.deb ... Unpacking libtool (2.4.7-5) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../17-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../18-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libsub-override-perl. Preparing to unpack .../19-libsub-override-perl_0.09-4_all.deb ... Unpacking libsub-override-perl (0.09-4) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../20-libfile-stripnondeterminism-perl_1.13.1-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.13.1-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../21-dh-strip-nondeterminism_1.13.1-1_all.deb ... Unpacking dh-strip-nondeterminism (1.13.1-1) ... Selecting previously unselected package libelf1:amd64. Preparing to unpack .../22-libelf1_0.188-2.1_amd64.deb ... Unpacking libelf1:amd64 (0.188-2.1) ... Selecting previously unselected package dwz. Preparing to unpack .../23-dwz_0.15-1_amd64.deb ... Unpacking dwz (0.15-1) ... Selecting previously unselected package libicu72:amd64. Preparing to unpack .../24-libicu72_72.1-3_amd64.deb ... Unpacking libicu72:amd64 (72.1-3) ... Selecting previously unselected package libxml2:amd64. Preparing to unpack .../25-libxml2_2.9.14+dfsg-1.3~deb12u1_amd64.deb ... Unpacking libxml2:amd64 (2.9.14+dfsg-1.3~deb12u1) ... Selecting previously unselected package gettext. Preparing to unpack .../26-gettext_0.21-12_amd64.deb ... Unpacking gettext (0.21-12) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../27-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../28-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../29-debhelper_13.11.4_all.deb ... Unpacking debhelper (13.11.4) ... Selecting previously unselected package dh-golang. Preparing to unpack .../30-dh-golang_1.59_all.deb ... Unpacking dh-golang (1.59) ... Selecting previously unselected package golang-1.19-src. Preparing to unpack .../31-golang-1.19-src_1.19.8-2_all.deb ... Unpacking golang-1.19-src (1.19.8-2) ... Selecting previously unselected package golang-1.19-go. Preparing to unpack .../32-golang-1.19-go_1.19.8-2_amd64.deb ... Unpacking golang-1.19-go (1.19.8-2) ... Selecting previously unselected package golang-src. Preparing to unpack .../33-golang-src_2%3a1.19~1_all.deb ... Unpacking golang-src (2:1.19~1) ... Selecting previously unselected package golang-go:amd64. Preparing to unpack .../34-golang-go_2%3a1.19~1_amd64.deb ... Unpacking golang-go:amd64 (2:1.19~1) ... Selecting previously unselected package golang-any:amd64. Preparing to unpack .../35-golang-any_2%3a1.19~1_amd64.deb ... Unpacking golang-any:amd64 (2:1.19~1) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../36-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../37-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../38-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../39-golang-golang-x-sys-dev_0.3.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.3.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../40-golang-github-mattn-go-isatty-dev_0.0.17-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.17-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../41-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../42-golang-github-fatih-color-dev_1.7.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.7.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../43-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../44-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../45-golang-github-stretchr-objx-dev_0.5.0-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../46-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../47-golang-github-stretchr-testify-dev_1.8.1-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.8.1-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../48-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../49-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../50-golang-github-pkg-errors-dev_0.9.1-2_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-2) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../51-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../52-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../53-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../54-golang-github-hashicorp-go-uuid-dev_1.0.2-2_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.2-2) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../55-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../56-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../57-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../58-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../59-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ... Selecting previously unselected package golang-github-golang-protobuf-1-3-dev. Preparing to unpack .../60-golang-github-golang-protobuf-1-3-dev_1.3.5-4_all.deb ... Unpacking golang-github-golang-protobuf-1-3-dev (1.3.5-4) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../61-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../62-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../63-golang-github-json-iterator-go-dev_1.1.12-1_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-1) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../64-golang-github-prometheus-client-model-dev_0.3.0-3_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.3.0-3) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../65-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../66-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../67-golang-golang-x-text-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.7.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../68-golang-golang-x-net-dev_1%3a0.7.0+dfsg-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.7.0+dfsg-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../69-golang-github-aws-aws-sdk-go-dev_1.44.133-1_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.44.133-1) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../70-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../71-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../72-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../73-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../74-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../75-golang-golang-x-oauth2-dev_0.3.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.3.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../76-golang-google-protobuf-dev_1.28.1-3_all.deb ... Unpacking golang-google-protobuf-dev (1.28.1-3) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../77-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-protobuf-extensions-dev. Preparing to unpack .../78-golang-protobuf-extensions-dev_1.0.1-3_all.deb ... Unpacking golang-protobuf-extensions-dev (1.0.1-3) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../79-golang-github-prometheus-common-dev_0.39.0-2_all.deb ... Unpacking golang-github-prometheus-common-dev (0.39.0-2) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../80-golang-golang-x-sync-dev_0.1.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.1.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../81-golang-github-prometheus-procfs-dev_0.8.0-3_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.8.0-3) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../82-golang-github-prometheus-client-golang-dev_1.14.0-3_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.14.0-3) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../83-golang-github-armon-go-metrics-dev_0.3.4-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.3.4-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../84-golang-github-creack-pty-dev_1.1.18-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.18-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../85-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../86-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../87-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:amd64. Preparing to unpack .../88-libsasl2-modules-db_2.1.28+dfsg-10_amd64.deb ... Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg-10) ... Selecting previously unselected package libsasl2-2:amd64. Preparing to unpack .../89-libsasl2-2_2.1.28+dfsg-10_amd64.deb ... Unpacking libsasl2-2:amd64 (2.1.28+dfsg-10) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../90-libsasl2-dev_2.1.28+dfsg-10_amd64.deb ... Unpacking libsasl2-dev (2.1.28+dfsg-10) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../91-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-github-pkg-diff-dev. Preparing to unpack .../92-golang-github-pkg-diff-dev_0.0~git20210226.20ebb0f-1_all.deb ... Unpacking golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../93-golang-github-rogpeppe-go-internal-dev_1.9.0-1_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.9.0-1) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../94-golang-github-kr-pretty-dev_0.3.0-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.0-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../95-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../96-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../97-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:amd64 (1.5.7-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.2-2) ... Setting up libicu72:amd64 (72.1-3) ... Setting up bsdextrautils (2.38.1-5+b1) ... Setting up golang-github-creack-pty-dev (1.1.18-1) ... Setting up golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ... Setting up golang-github-stretchr-objx-dev (0.5.0-1) ... Setting up libmagic-mgc (1:5.44-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-2) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.11.4) ... Setting up golang-golang-x-sys-dev (0.3.0-1) ... Setting up libmagic1:amd64 (1:5.44-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-12) ... Setting up golang-1.19-src (1.19.8-2) ... Setting up m4 (1.4.19-3) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-github-golang-protobuf-1-3-dev (1.3.5-4) ... Setting up file (1:5.44-3) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg-10) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.28.1-3) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.8.1-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-golang-x-sync-dev (0.1.0-1) ... Setting up autopoint (0.21-12) ... Setting up libsasl2-2:amd64 (2.1.28+dfsg-10) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.71-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up sensible-utils (0.0.17+nmu1) ... Setting up golang-golang-x-text-dev (0.7.0-1) ... Setting up libuchardet0:amd64 (0.0.7-1) ... Setting up golang-github-prometheus-procfs-dev (0.8.0-3) ... Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up libsub-override-perl (0.09-4) ... Setting up golang-github-prometheus-client-model-dev (0.3.0-3) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-github-rogpeppe-go-internal-dev (1.9.0-1) ... Setting up golang-src (2:1.19~1) ... Setting up libelf1:amd64 (0.188-2.1) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:amd64 (2.9.14+dfsg-1.3~deb12u1) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up golang-1.19-go (1.19.8-2) ... Setting up automake (1:1.16.5-1.3) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up golang-github-kr-pretty-dev (0.3.0-1) ... Setting up libfile-stripnondeterminism-perl (1.13.1-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.17-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up gettext (0.21-12) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.4.7-5) ... Setting up golang-go:amd64 (2:1.19~1) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-protobuf-extensions-dev (1.0.1-3) ... Setting up golang-github-json-iterator-go-dev (1.1.12-1) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-any:amd64 (2:1.19~1) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg-10) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-golang-x-net-dev (1:0.7.0+dfsg-1) ... Setting up dh-strip-nondeterminism (1.13.1-1) ... Setting up dwz (0.15-1) ... Setting up groff-base (1.22.4-10) ... Setting up golang-golang-x-oauth2-dev (0.3.0-1) ... Setting up golang-github-aws-aws-sdk-go-dev (1.44.133-1) ... Setting up golang-github-fatih-color-dev (1.7.0-1) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up man-db (2.11.2-2) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up debhelper (13.11.4) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.59) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Setting up golang-github-prometheus-common-dev (0.39.0-2) ... Setting up golang-github-prometheus-client-golang-dev (1.14.0-3) ... Setting up golang-github-armon-go-metrics-dev (0.3.4-1) ... Processing triggers for libc-bin (2.36-9+deb12u3) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps Reading package lists... Building dependency tree... Reading state information... fakeroot is already the newest version (1.31-1.2). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . dpkg-buildpackage: info: host architecture amd64 fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build dh build --buildsystem=golang --with=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-x86_64-linux-gnu && go install -trimpath -v -p 16 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/goarch crypto/subtle math/bits internal/unsafeheader internal/goos internal/race internal/cpu runtime/internal/syscall internal/goexperiment crypto/internal/subtle container/list unicode/utf8 sync/atomic internal/itoa unicode internal/abi runtime/internal/atomic runtime/internal/math runtime/internal/sys crypto/internal/boring/sig encoding unicode/utf16 internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle internal/bytealg math runtime internal/reflectlite sync internal/testlog math/rand internal/singleflight internal/sysinfo runtime/cgo errors sort internal/oserror io internal/safefilepath path github.com/hashicorp/golang-lru/simplelru vendor/golang.org/x/net/dns/dnsmessage strconv crypto/internal/nistec/fiat syscall bytes strings crypto/internal/randutil hash hash/crc64 hash/crc32 vendor/golang.org/x/text/transform crypto crypto/rc4 reflect bufio github.com/hashicorp/go-immutable-radix net/http/internal/ascii html regexp/syntax internal/syscall/unix internal/syscall/execenv time regexp context internal/poll io/fs embed os internal/fmtsort encoding/binary encoding/base64 crypto/md5 vendor/golang.org/x/crypto/internal/poly1305 crypto/internal/edwards25519/field crypto/cipher crypto/internal/nistec vendor/golang.org/x/crypto/curve25519/internal/field golang.org/x/sys/unix encoding/pem internal/godebug io/ioutil fmt path/filepath os/signal crypto/internal/edwards25519 internal/intern vendor/golang.org/x/sys/cpu crypto/internal/boring crypto/des vendor/golang.org/x/crypto/chacha20 net/netip crypto/aes crypto/sha1 crypto/hmac crypto/sha256 crypto/sha512 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/crypto/chacha20poly1305 net log encoding/json encoding/hex net/url vendor/golang.org/x/crypto/curve25519 mime/quotedprintable compress/flate mime vendor/golang.org/x/net/http2/hpack math/big net/http/internal vendor/golang.org/x/text/unicode/norm go/token encoding/gob text/template/parse flag runtime/debug runtime/trace vendor/golang.org/x/text/unicode/bidi testing compress/gzip github.com/mattn/go-isatty vendor/golang.org/x/text/secure/bidirule github.com/mattn/go-colorable github.com/fatih/color text/template github.com/hashicorp/go-hclog vendor/golang.org/x/net/idna crypto/rand crypto/internal/boring/bbig crypto/elliptic encoding/asn1 crypto/dsa crypto/ed25519 crypto/rsa html/template crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte crypto/ecdsa vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang cd obj-x86_64-linux-gnu && go test -vet=off -v -p 16 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir testing.go:153: 2025-02-09T16:52:05.674Z [INFO] creating new snapshot: path=/tmp/raft2124179462/raft1841833666/snapshots/3-10-1739119925674.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot testing.go:153: 2025-02-09T16:52:05.676Z [INFO] creating new snapshot: path=/tmp/raft2278030043/snapshots/3-10-1739119925676.tmp testing.go:153: 2025-02-09T16:52:05.676Z [WARN] found temporary snapshot: name=3-10-1739119925676.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot testing.go:153: 2025-02-09T16:52:05.677Z [INFO] creating new snapshot: path=/tmp/raft4100629497/snapshots/3-10-1739119925677.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention testing.go:153: 2025-02-09T16:52:05.677Z [INFO] creating new snapshot: path=/tmp/raft2699834459/snapshots/3-10-1739119925677.tmp testing.go:153: 2025-02-09T16:52:05.677Z [INFO] creating new snapshot: path=/tmp/raft2699834459/snapshots/3-11-1739119925677.tmp testing.go:153: 2025-02-09T16:52:05.677Z [INFO] creating new snapshot: path=/tmp/raft2699834459/snapshots/3-12-1739119925677.tmp testing.go:153: 2025-02-09T16:52:05.678Z [INFO] reaping snapshot: path=/tmp/raft2699834459/snapshots/3-10-1739119925677 testing.go:153: 2025-02-09T16:52:05.678Z [INFO] creating new snapshot: path=/tmp/raft2699834459/snapshots/3-13-1739119925678.tmp testing.go:153: 2025-02-09T16:52:05.678Z [INFO] reaping snapshot: path=/tmp/raft2699834459/snapshots/3-11-1739119925677 testing.go:153: 2025-02-09T16:52:05.678Z [INFO] creating new snapshot: path=/tmp/raft2699834459/snapshots/3-14-1739119925678.tmp testing.go:153: 2025-02-09T16:52:05.679Z [INFO] reaping snapshot: path=/tmp/raft2699834459/snapshots/3-12-1739119925677 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering testing.go:153: 2025-02-09T16:52:05.680Z [INFO] creating new snapshot: path=/tmp/raft1835152318/snapshots/5-130350-1739119925680.tmp testing.go:153: 2025-02-09T16:52:05.680Z [INFO] creating new snapshot: path=/tmp/raft1835152318/snapshots/36-204917-1739119925680.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:718: countingWriter: 2025-02-09T16:52:05.702Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T16:52:05.707Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T16:52:05.717Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T16:52:05.738Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T16:52:05.778Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T16:52:05.859Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T16:52:06.020Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T16:52:06.341Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T16:52:06.982Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T16:52:07.986Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T16:52:08.990Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-264fd0ef-0069-9cdc-025b-8a5a1322e172: 2025-02-09T16:52:09.703Z [INFO] server-264fd0ef-0069-9cdc-025b-8a5a1322e172: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-264fd0ef-0069-9cdc-025b-8a5a1322e172 Address:264fd0ef-0069-9cdc-025b-8a5a1322e172}]" testing.go:149: server-264fd0ef-0069-9cdc-025b-8a5a1322e172: 2025-02-09T16:52:09.703Z [INFO] server-264fd0ef-0069-9cdc-025b-8a5a1322e172: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-264fd0ef-0069-9cdc-025b-8a5a1322e172 Address:264fd0ef-0069-9cdc-025b-8a5a1322e172}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-bba6e73c-524c-fb57-7523-2f4923854f31: 2025-02-09T16:52:09.704Z [INFO] server-bba6e73c-524c-fb57-7523-2f4923854f31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bba6e73c-524c-fb57-7523-2f4923854f31 Address:bba6e73c-524c-fb57-7523-2f4923854f31}]" testing.go:149: server-bba6e73c-524c-fb57-7523-2f4923854f31: 2025-02-09T16:52:09.704Z [INFO] server-bba6e73c-524c-fb57-7523-2f4923854f31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bba6e73c-524c-fb57-7523-2f4923854f31 Address:bba6e73c-524c-fb57-7523-2f4923854f31}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.704Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: initial configuration: index=0 servers=[] testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.704Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: 2025-02-09T16:52:09.704Z [INFO] server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: initial configuration: index=0 servers=[] testing.go:149: server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: 2025-02-09T16:52:09.704Z [INFO] server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: 2025-02-09T16:52:09.704Z [INFO] server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: initial configuration: index=0 servers=[] testing.go:149: server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: 2025-02-09T16:52:09.704Z [INFO] server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.704Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: entering follower state: follower="Node at 5f15be0d-9a7e-ffce-01fb-83cbe18012a1 [Follower]" leader= testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.704Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: entering follower state: follower="Node at 5f15be0d-9a7e-ffce-01fb-83cbe18012a1 [Follower]" leader= testLoggerAdapter verbose: server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: 2025-02-09T16:52:09.704Z [INFO] server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: entering follower state: follower="Node at 5700b5ea-fb27-fbd9-5ab2-8268ffc31035 [Follower]" leader= testing.go:149: server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: 2025-02-09T16:52:09.704Z [INFO] server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: entering follower state: follower="Node at 5700b5ea-fb27-fbd9-5ab2-8268ffc31035 [Follower]" leader= testLoggerAdapter verbose: cluster: 2025-02-09T16:52:09.704Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:09.704Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: 2025-02-09T16:52:09.704Z [INFO] server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: entering follower state: follower="Node at f252f6d4-f1c3-238b-e28e-557bd991a2ec [Follower]" leader= testing.go:149: server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: 2025-02-09T16:52:09.704Z [INFO] server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: entering follower state: follower="Node at f252f6d4-f1c3-238b-e28e-557bd991a2ec [Follower]" leader= testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [WARN] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [WARN] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: entering candidate state: node="Node at 5f15be0d-9a7e-ffce-01fb-83cbe18012a1 [Candidate]" term=2 testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: entering candidate state: node="Node at 5f15be0d-9a7e-ffce-01fb-83cbe18012a1 [Candidate]" term=2 testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: election won: tally=2 testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: election won: tally=2 testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: entering leader state: leader="Node at 5f15be0d-9a7e-ffce-01fb-83cbe18012a1 [Leader]" testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: entering leader state: leader="Node at 5f15be0d-9a7e-ffce-01fb-83cbe18012a1 [Leader]" testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: added peer, starting replication: peer=server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035 testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: added peer, starting replication: peer=server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035 testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: added peer, starting replication: peer=server-f252f6d4-f1c3-238b-e28e-557bd991a2ec testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: added peer, starting replication: peer=server-f252f6d4-f1c3-238b-e28e-557bd991a2ec testLoggerAdapter verbose: server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: 2025-02-09T16:52:09.756Z [WARN] server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: 2025-02-09T16:52:09.756Z [WARN] server-f252f6d4-f1c3-238b-e28e-557bd991a2ec: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: 2025-02-09T16:52:09.756Z [WARN] server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: 2025-02-09T16:52:09.756Z [WARN] server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [WARN] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: appendEntries rejected, sending older logs: peer="{Voter server-f252f6d4-f1c3-238b-e28e-557bd991a2ec f252f6d4-f1c3-238b-e28e-557bd991a2ec}" next=1 testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [WARN] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: appendEntries rejected, sending older logs: peer="{Voter server-f252f6d4-f1c3-238b-e28e-557bd991a2ec f252f6d4-f1c3-238b-e28e-557bd991a2ec}" next=1 testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [WARN] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: appendEntries rejected, sending older logs: peer="{Voter server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035 5700b5ea-fb27-fbd9-5ab2-8268ffc31035}" next=1 testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [WARN] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: appendEntries rejected, sending older logs: peer="{Voter server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035 5700b5ea-fb27-fbd9-5ab2-8268ffc31035}" next=1 testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: pipelining replication: peer="{Voter server-f252f6d4-f1c3-238b-e28e-557bd991a2ec f252f6d4-f1c3-238b-e28e-557bd991a2ec}" testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: pipelining replication: peer="{Voter server-f252f6d4-f1c3-238b-e28e-557bd991a2ec f252f6d4-f1c3-238b-e28e-557bd991a2ec}" testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: pipelining replication: peer="{Voter server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035 5700b5ea-fb27-fbd9-5ab2-8268ffc31035}" testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.756Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: pipelining replication: peer="{Voter server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035 5700b5ea-fb27-fbd9-5ab2-8268ffc31035}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:09.861Z [INFO] stable state for Follower reached at 2025-02-09 16:52:09.756555682 +0000 UTC m=+4.086788370 (2 nodes), 51.61854ms from start of poll, 52.122415ms from cluster start. Timeout at 2025-02-09 16:52:09.86170084 +0000 UTC m=+4.191933526, 105.145156ms after stability testing.go:149: cluster: 2025-02-09T16:52:09.861Z [INFO] stable state for Follower reached at 2025-02-09 16:52:09.756555682 +0000 UTC m=+4.086788370 (2 nodes), 51.61854ms from start of poll, 52.122415ms from cluster start. Timeout at 2025-02-09 16:52:09.86170084 +0000 UTC m=+4.191933526, 105.145156ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:09.861Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:09.861Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:09.967Z [INFO] stable state for Leader reached at 2025-02-09 16:52:09.861901531 +0000 UTC m=+4.192134217 (1 nodes), 1.073µs from start of poll, 157.468262ms from cluster start. Timeout at 2025-02-09 16:52:09.967655742 +0000 UTC m=+4.297888428, 105.754211ms after stability testing.go:149: cluster: 2025-02-09T16:52:09.967Z [INFO] stable state for Leader reached at 2025-02-09 16:52:09.861901531 +0000 UTC m=+4.192134217 (1 nodes), 1.073µs from start of poll, 157.468262ms from cluster start. Timeout at 2025-02-09 16:52:09.967655742 +0000 UTC m=+4.297888428, 105.754211ms after stability testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.978Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: aborting pipeline replication: peer="{Voter server-f252f6d4-f1c3-238b-e28e-557bd991a2ec f252f6d4-f1c3-238b-e28e-557bd991a2ec}" testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.978Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: aborting pipeline replication: peer="{Voter server-f252f6d4-f1c3-238b-e28e-557bd991a2ec f252f6d4-f1c3-238b-e28e-557bd991a2ec}" testLoggerAdapter verbose: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.978Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: aborting pipeline replication: peer="{Voter server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035 5700b5ea-fb27-fbd9-5ab2-8268ffc31035}" testing.go:149: server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: 2025-02-09T16:52:09.978Z [INFO] server-5f15be0d-9a7e-ffce-01fb-83cbe18012a1: aborting pipeline replication: peer="{Voter server-5700b5ea-fb27-fbd9-5ab2-8268ffc31035 5700b5ea-fb27-fbd9-5ab2-8268ffc31035}" --- PASS: TestRaft_LiveBootstrap (0.28s) === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-b00b061c-871f-9831-cadc-a8574e2c6ffa: 2025-02-09T16:52:09.979Z [INFO] server-b00b061c-871f-9831-cadc-a8574e2c6ffa: initial configuration: index=0 servers=[] testing.go:149: server-b00b061c-871f-9831-cadc-a8574e2c6ffa: 2025-02-09T16:52:09.979Z [INFO] server-b00b061c-871f-9831-cadc-a8574e2c6ffa: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:09.980Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad1a75c9-d83e-a691-a449-1d30094956fb Address:ad1a75c9-d83e-a691-a449-1d30094956fb} {Suffrage:Voter ID:server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 Address:2d2d5767-a1d3-dde9-0c42-75588c9761b7}]" testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:09.980Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad1a75c9-d83e-a691-a449-1d30094956fb Address:ad1a75c9-d83e-a691-a449-1d30094956fb} {Suffrage:Voter ID:server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 Address:2d2d5767-a1d3-dde9-0c42-75588c9761b7}]" testLoggerAdapter verbose: server-2d2d5767-a1d3-dde9-0c42-75588c9761b7: 2025-02-09T16:52:09.980Z [INFO] server-2d2d5767-a1d3-dde9-0c42-75588c9761b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad1a75c9-d83e-a691-a449-1d30094956fb Address:ad1a75c9-d83e-a691-a449-1d30094956fb} {Suffrage:Voter ID:server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 Address:2d2d5767-a1d3-dde9-0c42-75588c9761b7}]" testing.go:149: server-2d2d5767-a1d3-dde9-0c42-75588c9761b7: 2025-02-09T16:52:09.980Z [INFO] server-2d2d5767-a1d3-dde9-0c42-75588c9761b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad1a75c9-d83e-a691-a449-1d30094956fb Address:ad1a75c9-d83e-a691-a449-1d30094956fb} {Suffrage:Voter ID:server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 Address:2d2d5767-a1d3-dde9-0c42-75588c9761b7}]" testLoggerAdapter verbose: server-7067d9d4-3510-167b-0d2c-083d830f8bdb: 2025-02-09T16:52:09.980Z [INFO] server-7067d9d4-3510-167b-0d2c-083d830f8bdb: initial configuration: index=0 servers=[] testing.go:149: server-7067d9d4-3510-167b-0d2c-083d830f8bdb: 2025-02-09T16:52:09.980Z [INFO] server-7067d9d4-3510-167b-0d2c-083d830f8bdb: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2025-02-09T16:52:09.980Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:09.980Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:09.980Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: entering follower state: follower="Node at ad1a75c9-d83e-a691-a449-1d30094956fb [Follower]" leader= testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:09.980Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: entering follower state: follower="Node at ad1a75c9-d83e-a691-a449-1d30094956fb [Follower]" leader= testLoggerAdapter verbose: server-2d2d5767-a1d3-dde9-0c42-75588c9761b7: 2025-02-09T16:52:09.980Z [INFO] server-2d2d5767-a1d3-dde9-0c42-75588c9761b7: entering follower state: follower="Node at 2d2d5767-a1d3-dde9-0c42-75588c9761b7 [Follower]" leader= testing.go:149: server-2d2d5767-a1d3-dde9-0c42-75588c9761b7: 2025-02-09T16:52:09.980Z [INFO] server-2d2d5767-a1d3-dde9-0c42-75588c9761b7: entering follower state: follower="Node at 2d2d5767-a1d3-dde9-0c42-75588c9761b7 [Follower]" leader= testLoggerAdapter verbose: server-7067d9d4-3510-167b-0d2c-083d830f8bdb: 2025-02-09T16:52:09.980Z [INFO] server-7067d9d4-3510-167b-0d2c-083d830f8bdb: entering follower state: follower="Node at 7067d9d4-3510-167b-0d2c-083d830f8bdb [Follower]" leader= testing.go:149: server-7067d9d4-3510-167b-0d2c-083d830f8bdb: 2025-02-09T16:52:09.980Z [INFO] server-7067d9d4-3510-167b-0d2c-083d830f8bdb: entering follower state: follower="Node at 7067d9d4-3510-167b-0d2c-083d830f8bdb [Follower]" leader= testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.034Z [WARN] server-ad1a75c9-d83e-a691-a449-1d30094956fb: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.034Z [WARN] server-ad1a75c9-d83e-a691-a449-1d30094956fb: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.034Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: entering candidate state: node="Node at ad1a75c9-d83e-a691-a449-1d30094956fb [Candidate]" term=2 testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.034Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: entering candidate state: node="Node at ad1a75c9-d83e-a691-a449-1d30094956fb [Candidate]" term=2 testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.035Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: election won: tally=2 testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.035Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: election won: tally=2 testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.035Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: entering leader state: leader="Node at ad1a75c9-d83e-a691-a449-1d30094956fb [Leader]" testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.035Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: entering leader state: leader="Node at ad1a75c9-d83e-a691-a449-1d30094956fb [Leader]" testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.035Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: added peer, starting replication: peer=server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.035Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: added peer, starting replication: peer=server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.035Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: pipelining replication: peer="{Voter server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 2d2d5767-a1d3-dde9-0c42-75588c9761b7}" testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.035Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: pipelining replication: peer="{Voter server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 2d2d5767-a1d3-dde9-0c42-75588c9761b7}" testLoggerAdapter verbose: server-7067d9d4-3510-167b-0d2c-083d830f8bdb: 2025-02-09T16:52:10.043Z [WARN] server-7067d9d4-3510-167b-0d2c-083d830f8bdb: no known peers, aborting election testing.go:149: server-7067d9d4-3510-167b-0d2c-083d830f8bdb: 2025-02-09T16:52:10.043Z [WARN] server-7067d9d4-3510-167b-0d2c-083d830f8bdb: no known peers, aborting election testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.140Z [INFO] stable state for Leader reached at 2025-02-09 16:52:10.035132514 +0000 UTC m=+4.365365206 (1 nodes), 54.598136ms from start of poll, 55.039444ms from cluster start. Timeout at 2025-02-09 16:52:10.140504997 +0000 UTC m=+4.470737683, 105.372477ms after stability testing.go:149: cluster: 2025-02-09T16:52:10.140Z [INFO] stable state for Leader reached at 2025-02-09 16:52:10.035132514 +0000 UTC m=+4.365365206 (1 nodes), 54.598136ms from start of poll, 55.039444ms from cluster start. Timeout at 2025-02-09 16:52:10.140504997 +0000 UTC m=+4.470737683, 105.372477ms after stability testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.140Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: updating configuration: command=AddStaging server-id=server-7067d9d4-3510-167b-0d2c-083d830f8bdb server-addr=7067d9d4-3510-167b-0d2c-083d830f8bdb servers="[{Suffrage:Voter ID:server-ad1a75c9-d83e-a691-a449-1d30094956fb Address:ad1a75c9-d83e-a691-a449-1d30094956fb} {Suffrage:Voter ID:server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 Address:2d2d5767-a1d3-dde9-0c42-75588c9761b7} {Suffrage:Voter ID:server-7067d9d4-3510-167b-0d2c-083d830f8bdb Address:7067d9d4-3510-167b-0d2c-083d830f8bdb}]" testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.140Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: updating configuration: command=AddStaging server-id=server-7067d9d4-3510-167b-0d2c-083d830f8bdb server-addr=7067d9d4-3510-167b-0d2c-083d830f8bdb servers="[{Suffrage:Voter ID:server-ad1a75c9-d83e-a691-a449-1d30094956fb Address:ad1a75c9-d83e-a691-a449-1d30094956fb} {Suffrage:Voter ID:server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 Address:2d2d5767-a1d3-dde9-0c42-75588c9761b7} {Suffrage:Voter ID:server-7067d9d4-3510-167b-0d2c-083d830f8bdb Address:7067d9d4-3510-167b-0d2c-083d830f8bdb}]" testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.140Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: added peer, starting replication: peer=server-7067d9d4-3510-167b-0d2c-083d830f8bdb testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.140Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: added peer, starting replication: peer=server-7067d9d4-3510-167b-0d2c-083d830f8bdb testLoggerAdapter verbose: server-7067d9d4-3510-167b-0d2c-083d830f8bdb: 2025-02-09T16:52:10.140Z [WARN] server-7067d9d4-3510-167b-0d2c-083d830f8bdb: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-7067d9d4-3510-167b-0d2c-083d830f8bdb: 2025-02-09T16:52:10.140Z [WARN] server-7067d9d4-3510-167b-0d2c-083d830f8bdb: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.140Z [WARN] server-ad1a75c9-d83e-a691-a449-1d30094956fb: appendEntries rejected, sending older logs: peer="{Voter server-7067d9d4-3510-167b-0d2c-083d830f8bdb 7067d9d4-3510-167b-0d2c-083d830f8bdb}" next=1 testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.140Z [WARN] server-ad1a75c9-d83e-a691-a449-1d30094956fb: appendEntries rejected, sending older logs: peer="{Voter server-7067d9d4-3510-167b-0d2c-083d830f8bdb 7067d9d4-3510-167b-0d2c-083d830f8bdb}" next=1 testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.140Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: pipelining replication: peer="{Voter server-7067d9d4-3510-167b-0d2c-083d830f8bdb 7067d9d4-3510-167b-0d2c-083d830f8bdb}" testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.140Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: pipelining replication: peer="{Voter server-7067d9d4-3510-167b-0d2c-083d830f8bdb 7067d9d4-3510-167b-0d2c-083d830f8bdb}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.145Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:10.145Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.251Z [INFO] stable state for Leader reached at 2025-02-09 16:52:10.146036085 +0000 UTC m=+4.476268771 (1 nodes), 1.101µs from start of poll, 165.943009ms from cluster start. Timeout at 2025-02-09 16:52:10.251145496 +0000 UTC m=+4.581378192, 105.109421ms after stability testing.go:149: cluster: 2025-02-09T16:52:10.251Z [INFO] stable state for Leader reached at 2025-02-09 16:52:10.146036085 +0000 UTC m=+4.476268771 (1 nodes), 1.101µs from start of poll, 165.943009ms from cluster start. Timeout at 2025-02-09 16:52:10.251145496 +0000 UTC m=+4.581378192, 105.109421ms after stability testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.251Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: aborting pipeline replication: peer="{Voter server-7067d9d4-3510-167b-0d2c-083d830f8bdb 7067d9d4-3510-167b-0d2c-083d830f8bdb}" testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.251Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: aborting pipeline replication: peer="{Voter server-7067d9d4-3510-167b-0d2c-083d830f8bdb 7067d9d4-3510-167b-0d2c-083d830f8bdb}" testLoggerAdapter verbose: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.251Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: aborting pipeline replication: peer="{Voter server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 2d2d5767-a1d3-dde9-0c42-75588c9761b7}" testing.go:149: server-ad1a75c9-d83e-a691-a449-1d30094956fb: 2025-02-09T16:52:10.251Z [INFO] server-ad1a75c9-d83e-a691-a449-1d30094956fb: aborting pipeline replication: peer="{Voter server-2d2d5767-a1d3-dde9-0c42-75588c9761b7 2d2d5767-a1d3-dde9-0c42-75588c9761b7}" --- PASS: TestRaft_HasExistingState (0.27s) === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.252Z [INFO] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3 Address:1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3}]" testing.go:149: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.252Z [INFO] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3 Address:1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3}]" testLoggerAdapter verbose: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.252Z [INFO] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: entering follower state: follower="Node at 1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3 [Follower]" leader= testing.go:149: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.252Z [INFO] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: entering follower state: follower="Node at 1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3 [Follower]" leader= testLoggerAdapter verbose: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.304Z [WARN] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.304Z [WARN] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.304Z [INFO] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: entering candidate state: node="Node at 1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3 [Candidate]" term=2 testing.go:149: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.304Z [INFO] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: entering candidate state: node="Node at 1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3 [Candidate]" term=2 testLoggerAdapter verbose: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.304Z [INFO] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: election won: tally=1 testing.go:149: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.304Z [INFO] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: election won: tally=1 testLoggerAdapter verbose: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.304Z [INFO] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: entering leader state: leader="Node at 1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3 [Leader]" testing.go:149: server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: 2025-02-09T16:52:10.304Z [INFO] server-1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3: entering leader state: leader="Node at 1ba8c490-c839-52c0-b31c-c4c5bcb4bcd3 [Leader]" --- PASS: TestRaft_SingleNode (0.05s) === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-835d1c4d-f59e-1010-3e1a-e4af53f529fd: 2025-02-09T16:52:10.305Z [INFO] server-835d1c4d-f59e-1010-3e1a-e4af53f529fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-835d1c4d-f59e-1010-3e1a-e4af53f529fd Address:835d1c4d-f59e-1010-3e1a-e4af53f529fd} {Suffrage:Voter ID:server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 Address:b33657ca-a0e4-91d9-cdd8-55e44e0e9f51} {Suffrage:Voter ID:server-5446f2a4-b763-8eae-9429-02a8f38af31d Address:5446f2a4-b763-8eae-9429-02a8f38af31d}]" testing.go:149: server-835d1c4d-f59e-1010-3e1a-e4af53f529fd: 2025-02-09T16:52:10.305Z [INFO] server-835d1c4d-f59e-1010-3e1a-e4af53f529fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-835d1c4d-f59e-1010-3e1a-e4af53f529fd Address:835d1c4d-f59e-1010-3e1a-e4af53f529fd} {Suffrage:Voter ID:server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 Address:b33657ca-a0e4-91d9-cdd8-55e44e0e9f51} {Suffrage:Voter ID:server-5446f2a4-b763-8eae-9429-02a8f38af31d Address:5446f2a4-b763-8eae-9429-02a8f38af31d}]" testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.305Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-835d1c4d-f59e-1010-3e1a-e4af53f529fd Address:835d1c4d-f59e-1010-3e1a-e4af53f529fd} {Suffrage:Voter ID:server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 Address:b33657ca-a0e4-91d9-cdd8-55e44e0e9f51} {Suffrage:Voter ID:server-5446f2a4-b763-8eae-9429-02a8f38af31d Address:5446f2a4-b763-8eae-9429-02a8f38af31d}]" testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.305Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-835d1c4d-f59e-1010-3e1a-e4af53f529fd Address:835d1c4d-f59e-1010-3e1a-e4af53f529fd} {Suffrage:Voter ID:server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 Address:b33657ca-a0e4-91d9-cdd8-55e44e0e9f51} {Suffrage:Voter ID:server-5446f2a4-b763-8eae-9429-02a8f38af31d Address:5446f2a4-b763-8eae-9429-02a8f38af31d}]" testLoggerAdapter verbose: server-5446f2a4-b763-8eae-9429-02a8f38af31d: 2025-02-09T16:52:10.305Z [INFO] server-5446f2a4-b763-8eae-9429-02a8f38af31d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-835d1c4d-f59e-1010-3e1a-e4af53f529fd Address:835d1c4d-f59e-1010-3e1a-e4af53f529fd} {Suffrage:Voter ID:server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 Address:b33657ca-a0e4-91d9-cdd8-55e44e0e9f51} {Suffrage:Voter ID:server-5446f2a4-b763-8eae-9429-02a8f38af31d Address:5446f2a4-b763-8eae-9429-02a8f38af31d}]" testing.go:149: server-5446f2a4-b763-8eae-9429-02a8f38af31d: 2025-02-09T16:52:10.305Z [INFO] server-5446f2a4-b763-8eae-9429-02a8f38af31d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-835d1c4d-f59e-1010-3e1a-e4af53f529fd Address:835d1c4d-f59e-1010-3e1a-e4af53f529fd} {Suffrage:Voter ID:server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 Address:b33657ca-a0e4-91d9-cdd8-55e44e0e9f51} {Suffrage:Voter ID:server-5446f2a4-b763-8eae-9429-02a8f38af31d Address:5446f2a4-b763-8eae-9429-02a8f38af31d}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.305Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:10.305Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-835d1c4d-f59e-1010-3e1a-e4af53f529fd: 2025-02-09T16:52:10.305Z [INFO] server-835d1c4d-f59e-1010-3e1a-e4af53f529fd: entering follower state: follower="Node at 835d1c4d-f59e-1010-3e1a-e4af53f529fd [Follower]" leader= testing.go:149: server-835d1c4d-f59e-1010-3e1a-e4af53f529fd: 2025-02-09T16:52:10.305Z [INFO] server-835d1c4d-f59e-1010-3e1a-e4af53f529fd: entering follower state: follower="Node at 835d1c4d-f59e-1010-3e1a-e4af53f529fd [Follower]" leader= testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.305Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: entering follower state: follower="Node at b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 [Follower]" leader= testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.305Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: entering follower state: follower="Node at b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 [Follower]" leader= testLoggerAdapter verbose: server-5446f2a4-b763-8eae-9429-02a8f38af31d: 2025-02-09T16:52:10.305Z [INFO] server-5446f2a4-b763-8eae-9429-02a8f38af31d: entering follower state: follower="Node at 5446f2a4-b763-8eae-9429-02a8f38af31d [Follower]" leader= testing.go:149: server-5446f2a4-b763-8eae-9429-02a8f38af31d: 2025-02-09T16:52:10.305Z [INFO] server-5446f2a4-b763-8eae-9429-02a8f38af31d: entering follower state: follower="Node at 5446f2a4-b763-8eae-9429-02a8f38af31d [Follower]" leader= testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.390Z [WARN] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.390Z [WARN] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: entering candidate state: node="Node at b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 [Candidate]" term=2 testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: entering candidate state: node="Node at b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 [Candidate]" term=2 testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: election won: tally=2 testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: election won: tally=2 testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: entering leader state: leader="Node at b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 [Leader]" testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: entering leader state: leader="Node at b33657ca-a0e4-91d9-cdd8-55e44e0e9f51 [Leader]" testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: added peer, starting replication: peer=server-835d1c4d-f59e-1010-3e1a-e4af53f529fd testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: added peer, starting replication: peer=server-835d1c4d-f59e-1010-3e1a-e4af53f529fd testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: added peer, starting replication: peer=server-5446f2a4-b763-8eae-9429-02a8f38af31d testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: added peer, starting replication: peer=server-5446f2a4-b763-8eae-9429-02a8f38af31d testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: pipelining replication: peer="{Voter server-5446f2a4-b763-8eae-9429-02a8f38af31d 5446f2a4-b763-8eae-9429-02a8f38af31d}" testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: pipelining replication: peer="{Voter server-5446f2a4-b763-8eae-9429-02a8f38af31d 5446f2a4-b763-8eae-9429-02a8f38af31d}" testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: pipelining replication: peer="{Voter server-835d1c4d-f59e-1010-3e1a-e4af53f529fd 835d1c4d-f59e-1010-3e1a-e4af53f529fd}" testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.391Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: pipelining replication: peer="{Voter server-835d1c4d-f59e-1010-3e1a-e4af53f529fd 835d1c4d-f59e-1010-3e1a-e4af53f529fd}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.496Z [INFO] stable state for Follower reached at 2025-02-09 16:52:10.391549977 +0000 UTC m=+4.721782666 (2 nodes), 85.74989ms from start of poll, 86.17866ms from cluster start. Timeout at 2025-02-09 16:52:10.496857165 +0000 UTC m=+4.827089851, 105.307185ms after stability testing.go:149: cluster: 2025-02-09T16:52:10.496Z [INFO] stable state for Follower reached at 2025-02-09 16:52:10.391549977 +0000 UTC m=+4.721782666 (2 nodes), 85.74989ms from start of poll, 86.17866ms from cluster start. Timeout at 2025-02-09 16:52:10.496857165 +0000 UTC m=+4.827089851, 105.307185ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.496Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:10.496Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.602Z [INFO] stable state for Leader reached at 2025-02-09 16:52:10.496975103 +0000 UTC m=+4.827207791 (1 nodes), 907ns from start of poll, 191.603785ms from cluster start. Timeout at 2025-02-09 16:52:10.602368314 +0000 UTC m=+4.932601000, 105.393209ms after stability testing.go:149: cluster: 2025-02-09T16:52:10.602Z [INFO] stable state for Leader reached at 2025-02-09 16:52:10.496975103 +0000 UTC m=+4.827207791 (1 nodes), 907ns from start of poll, 191.603785ms from cluster start. Timeout at 2025-02-09 16:52:10.602368314 +0000 UTC m=+4.932601000, 105.393209ms after stability testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.613Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: aborting pipeline replication: peer="{Voter server-5446f2a4-b763-8eae-9429-02a8f38af31d 5446f2a4-b763-8eae-9429-02a8f38af31d}" testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.613Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: aborting pipeline replication: peer="{Voter server-5446f2a4-b763-8eae-9429-02a8f38af31d 5446f2a4-b763-8eae-9429-02a8f38af31d}" testLoggerAdapter verbose: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.613Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: aborting pipeline replication: peer="{Voter server-835d1c4d-f59e-1010-3e1a-e4af53f529fd 835d1c4d-f59e-1010-3e1a-e4af53f529fd}" testing.go:149: server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: 2025-02-09T16:52:10.613Z [INFO] server-b33657ca-a0e4-91d9-cdd8-55e44e0e9f51: aborting pipeline replication: peer="{Voter server-835d1c4d-f59e-1010-3e1a-e4af53f529fd 835d1c4d-f59e-1010-3e1a-e4af53f529fd}" --- PASS: TestRaft_TripleNode (0.31s) === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:10.614Z [INFO] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3df739-60f3-04f4-cdd3-6aa793b08087 Address:ed3df739-60f3-04f4-cdd3-6aa793b08087} {Suffrage:Voter ID:server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a Address:c7bfc503-0499-6397-f90c-b7fe4bb4db0a} {Suffrage:Voter ID:server-77661ac0-09bb-b789-ba94-ff98cbd2fafc Address:77661ac0-09bb-b789-ba94-ff98cbd2fafc}]" testing.go:149: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:10.614Z [INFO] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3df739-60f3-04f4-cdd3-6aa793b08087 Address:ed3df739-60f3-04f4-cdd3-6aa793b08087} {Suffrage:Voter ID:server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a Address:c7bfc503-0499-6397-f90c-b7fe4bb4db0a} {Suffrage:Voter ID:server-77661ac0-09bb-b789-ba94-ff98cbd2fafc Address:77661ac0-09bb-b789-ba94-ff98cbd2fafc}]" testLoggerAdapter verbose: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:10.614Z [INFO] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: entering follower state: follower="Node at ed3df739-60f3-04f4-cdd3-6aa793b08087 [Follower]" leader= testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.614Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3df739-60f3-04f4-cdd3-6aa793b08087 Address:ed3df739-60f3-04f4-cdd3-6aa793b08087} {Suffrage:Voter ID:server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a Address:c7bfc503-0499-6397-f90c-b7fe4bb4db0a} {Suffrage:Voter ID:server-77661ac0-09bb-b789-ba94-ff98cbd2fafc Address:77661ac0-09bb-b789-ba94-ff98cbd2fafc}]" testing.go:149: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:10.614Z [INFO] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: entering follower state: follower="Node at ed3df739-60f3-04f4-cdd3-6aa793b08087 [Follower]" leader= testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.614Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3df739-60f3-04f4-cdd3-6aa793b08087 Address:ed3df739-60f3-04f4-cdd3-6aa793b08087} {Suffrage:Voter ID:server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a Address:c7bfc503-0499-6397-f90c-b7fe4bb4db0a} {Suffrage:Voter ID:server-77661ac0-09bb-b789-ba94-ff98cbd2fafc Address:77661ac0-09bb-b789-ba94-ff98cbd2fafc}]" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:10.614Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3df739-60f3-04f4-cdd3-6aa793b08087 Address:ed3df739-60f3-04f4-cdd3-6aa793b08087} {Suffrage:Voter ID:server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a Address:c7bfc503-0499-6397-f90c-b7fe4bb4db0a} {Suffrage:Voter ID:server-77661ac0-09bb-b789-ba94-ff98cbd2fafc Address:77661ac0-09bb-b789-ba94-ff98cbd2fafc}]" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:10.614Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3df739-60f3-04f4-cdd3-6aa793b08087 Address:ed3df739-60f3-04f4-cdd3-6aa793b08087} {Suffrage:Voter ID:server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a Address:c7bfc503-0499-6397-f90c-b7fe4bb4db0a} {Suffrage:Voter ID:server-77661ac0-09bb-b789-ba94-ff98cbd2fafc Address:77661ac0-09bb-b789-ba94-ff98cbd2fafc}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.614Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:10.614Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.614Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering follower state: follower="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Follower]" leader= testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.614Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering follower state: follower="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Follower]" leader= testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:10.614Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: entering follower state: follower="Node at 77661ac0-09bb-b789-ba94-ff98cbd2fafc [Follower]" leader= testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:10.614Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: entering follower state: follower="Node at 77661ac0-09bb-b789-ba94-ff98cbd2fafc [Follower]" leader= testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.675Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.675Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.675Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering candidate state: node="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Candidate]" term=2 testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.675Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering candidate state: node="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Candidate]" term=2 testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: election won: tally=2 testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: election won: tally=2 testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering leader state: leader="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Leader]" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering leader state: leader="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Leader]" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: added peer, starting replication: peer=server-ed3df739-60f3-04f4-cdd3-6aa793b08087 testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: added peer, starting replication: peer=server-ed3df739-60f3-04f4-cdd3-6aa793b08087 testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: added peer, starting replication: peer=server-77661ac0-09bb-b789-ba94-ff98cbd2fafc testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: added peer, starting replication: peer=server-77661ac0-09bb-b789-ba94-ff98cbd2fafc testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: pipelining replication: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: pipelining replication: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: pipelining replication: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.676Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: pipelining replication: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.781Z [INFO] stable state for Follower reached at 2025-02-09 16:52:10.676130966 +0000 UTC m=+5.006363654 (2 nodes), 61.250354ms from start of poll, 61.780636ms from cluster start. Timeout at 2025-02-09 16:52:10.78131815 +0000 UTC m=+5.111550838, 105.187184ms after stability testing.go:149: cluster: 2025-02-09T16:52:10.781Z [INFO] stable state for Follower reached at 2025-02-09 16:52:10.676130966 +0000 UTC m=+5.006363654 (2 nodes), 61.250354ms from start of poll, 61.780636ms from cluster start. Timeout at 2025-02-09 16:52:10.78131815 +0000 UTC m=+5.111550838, 105.187184ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.781Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:10.781Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.886Z [INFO] stable state for Leader reached at 2025-02-09 16:52:10.781447474 +0000 UTC m=+5.111680160 (1 nodes), 914ns from start of poll, 167.097142ms from cluster start. Timeout at 2025-02-09 16:52:10.886457094 +0000 UTC m=+5.216689780, 105.00962ms after stability testing.go:149: cluster: 2025-02-09T16:52:10.886Z [INFO] stable state for Leader reached at 2025-02-09 16:52:10.781447474 +0000 UTC m=+5.111680160 (1 nodes), 914ns from start of poll, 167.097142ms from cluster start. Timeout at 2025-02-09 16:52:10.886457094 +0000 UTC m=+5.216689780, 105.00962ms after stability raft_test.go:336: [INFO] Disconnecting Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Leader] testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.898Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=ed3df739-60f3-04f4-cdd3-6aa793b08087 error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.898Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=ed3df739-60f3-04f4-cdd3-6aa793b08087 error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.900Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to pipeline appendEntries: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="append pipeline closed" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.900Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to pipeline appendEntries: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="append pipeline closed" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.900Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: aborting pipeline replication: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.900Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: aborting pipeline replication: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:10.903Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:10.903Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.905Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to pipeline appendEntries: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="append pipeline closed" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.905Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to pipeline appendEntries: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="append pipeline closed" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.905Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: aborting pipeline replication: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.905Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: aborting pipeline replication: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.907Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=77661ac0-09bb-b789-ba94-ff98cbd2fafc error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.907Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=77661ac0-09bb-b789-ba94-ff98cbd2fafc error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.910Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.910Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.911Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.911Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.917Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=ed3df739-60f3-04f4-cdd3-6aa793b08087 error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.917Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=ed3df739-60f3-04f4-cdd3-6aa793b08087 error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.928Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=77661ac0-09bb-b789-ba94-ff98cbd2fafc error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.928Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=77661ac0-09bb-b789-ba94-ff98cbd2fafc error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.928Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.928Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.932Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.932Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.935Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=ed3df739-60f3-04f4-cdd3-6aa793b08087 error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.935Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=ed3df739-60f3-04f4-cdd3-6aa793b08087 error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.944Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.944Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.945Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to contact: server-id=server-ed3df739-60f3-04f4-cdd3-6aa793b08087 time=50.400081ms testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.945Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to contact: server-id=server-ed3df739-60f3-04f4-cdd3-6aa793b08087 time=50.400081ms testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.948Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=77661ac0-09bb-b789-ba94-ff98cbd2fafc error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.948Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=77661ac0-09bb-b789-ba94-ff98cbd2fafc error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.950Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.950Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.956Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to contact: server-id=server-ed3df739-60f3-04f4-cdd3-6aa793b08087 time=61.271419ms testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.956Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to contact: server-id=server-ed3df739-60f3-04f4-cdd3-6aa793b08087 time=61.271419ms testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.956Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to contact: server-id=server-77661ac0-09bb-b789-ba94-ff98cbd2fafc time=59.075846ms testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.956Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to contact: server-id=server-77661ac0-09bb-b789-ba94-ff98cbd2fafc time=59.075846ms testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.956Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to contact quorum of nodes, stepping down testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.956Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.956Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering follower state: follower="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Follower]" leader= testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.956Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering follower state: follower="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Follower]" leader= testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:10.960Z [WARN] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: heartbeat timeout reached, starting election: last-leader=c7bfc503-0499-6397-f90c-b7fe4bb4db0a testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:10.960Z [WARN] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: heartbeat timeout reached, starting election: last-leader=c7bfc503-0499-6397-f90c-b7fe4bb4db0a testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:10.961Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: entering candidate state: node="Node at 77661ac0-09bb-b789-ba94-ff98cbd2fafc [Candidate]" term=3 testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:10.961Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: entering candidate state: node="Node at 77661ac0-09bb-b789-ba94-ff98cbd2fafc [Candidate]" term=3 testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:10.961Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to make requestVote RPC: target="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:10.961Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to make requestVote RPC: target="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:10.961Z [WARN] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: rejecting vote request since we have a leader: from=77661ac0-09bb-b789-ba94-ff98cbd2fafc leader=c7bfc503-0499-6397-f90c-b7fe4bb4db0a testing.go:149: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:10.961Z [WARN] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: rejecting vote request since we have a leader: from=77661ac0-09bb-b789-ba94-ff98cbd2fafc leader=c7bfc503-0499-6397-f90c-b7fe4bb4db0a testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.967Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=ed3df739-60f3-04f4-cdd3-6aa793b08087 error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.967Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to heartbeat to: peer=ed3df739-60f3-04f4-cdd3-6aa793b08087 error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.970Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.970Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.977Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:10.977Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to appendEntries to: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testLoggerAdapter verbose: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:11.000Z [WARN] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: heartbeat timeout reached, starting election: last-leader=c7bfc503-0499-6397-f90c-b7fe4bb4db0a testing.go:149: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:11.000Z [WARN] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: heartbeat timeout reached, starting election: last-leader=c7bfc503-0499-6397-f90c-b7fe4bb4db0a testLoggerAdapter verbose: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:11.000Z [INFO] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: entering candidate state: node="Node at ed3df739-60f3-04f4-cdd3-6aa793b08087 [Candidate]" term=3 testing.go:149: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:11.000Z [INFO] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: entering candidate state: node="Node at ed3df739-60f3-04f4-cdd3-6aa793b08087 [Candidate]" term=3 testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.000Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: duplicate requestVote for same term: term=3 testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.000Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:11.000Z [ERROR] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: failed to make requestVote RPC: target="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:11.000Z [ERROR] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: failed to make requestVote RPC: target="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.042Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.042Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.043Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering candidate state: node="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Candidate]" term=3 testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.043Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering candidate state: node="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Candidate]" term=3 testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.043Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to make requestVote RPC: target="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.043Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to make requestVote RPC: target="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.043Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to make requestVote RPC: target="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.043Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to make requestVote RPC: target="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [WARN] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: Election timeout reached, restarting election testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [WARN] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: Election timeout reached, restarting election testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: entering candidate state: node="Node at 77661ac0-09bb-b789-ba94-ff98cbd2fafc [Candidate]" term=4 testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: entering candidate state: node="Node at 77661ac0-09bb-b789-ba94-ff98cbd2fafc [Candidate]" term=4 testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to make requestVote RPC: target="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to make requestVote RPC: target="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:11.048Z [INFO] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: entering follower state: follower="Node at ed3df739-60f3-04f4-cdd3-6aa793b08087 [Follower]" leader= testing.go:149: server-ed3df739-60f3-04f4-cdd3-6aa793b08087: 2025-02-09T16:52:11.048Z [INFO] server-ed3df739-60f3-04f4-cdd3-6aa793b08087: entering follower state: follower="Node at ed3df739-60f3-04f4-cdd3-6aa793b08087 [Follower]" leader= testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: election won: tally=2 testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: election won: tally=2 testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: entering leader state: leader="Node at 77661ac0-09bb-b789-ba94-ff98cbd2fafc [Leader]" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: entering leader state: leader="Node at 77661ac0-09bb-b789-ba94-ff98cbd2fafc [Leader]" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: added peer, starting replication: peer=server-ed3df739-60f3-04f4-cdd3-6aa793b08087 testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: added peer, starting replication: peer=server-ed3df739-60f3-04f4-cdd3-6aa793b08087 testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: added peer, starting replication: peer=server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: added peer, starting replication: peer=server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to appendEntries to: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to appendEntries to: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: pipelining replication: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.048Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: pipelining replication: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.055Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to heartbeat to: peer=c7bfc503-0499-6397-f90c-b7fe4bb4db0a error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.055Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to heartbeat to: peer=c7bfc503-0499-6397-f90c-b7fe4bb4db0a error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.069Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to appendEntries to: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.069Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to appendEntries to: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.074Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to heartbeat to: peer=c7bfc503-0499-6397-f90c-b7fe4bb4db0a error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.074Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to heartbeat to: peer=c7bfc503-0499-6397-f90c-b7fe4bb4db0a error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.089Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to appendEntries to: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.089Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to appendEntries to: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.094Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to heartbeat to: peer=c7bfc503-0499-6397-f90c-b7fe4bb4db0a error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.094Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to heartbeat to: peer=c7bfc503-0499-6397-f90c-b7fe4bb4db0a error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.099Z [WARN] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to contact: server-id=server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a time=50.559611ms testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.099Z [WARN] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to contact: server-id=server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a time=50.559611ms testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.119Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to appendEntries to: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.119Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to appendEntries to: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.122Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to heartbeat to: peer=c7bfc503-0499-6397-f90c-b7fe4bb4db0a error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.122Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to heartbeat to: peer=c7bfc503-0499-6397-f90c-b7fe4bb4db0a error="failed to connect to peer: c7bfc503-0499-6397-f90c-b7fe4bb4db0a" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.137Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: Election timeout reached, restarting election testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.137Z [WARN] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: Election timeout reached, restarting election testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.137Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering candidate state: node="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Candidate]" term=4 testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.137Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering candidate state: node="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Candidate]" term=4 testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.137Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to make requestVote RPC: target="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.137Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to make requestVote RPC: target="{Voter server-77661ac0-09bb-b789-ba94-ff98cbd2fafc 77661ac0-09bb-b789-ba94-ff98cbd2fafc}" error="failed to connect to peer: 77661ac0-09bb-b789-ba94-ff98cbd2fafc" testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.137Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to make requestVote RPC: target="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.137Z [ERROR] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: failed to make requestVote RPC: target="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" error="failed to connect to peer: ed3df739-60f3-04f4-cdd3-6aa793b08087" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.147Z [WARN] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to contact: server-id=server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a time=98.273173ms testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.147Z [WARN] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to contact: server-id=server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a time=98.273173ms testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.154Z [INFO] stable state for Leader reached at 2025-02-09 16:52:11.04890515 +0000 UTC m=+5.379137838 (1 nodes), 145.443908ms from start of poll, 434.55482ms from cluster start. Timeout at 2025-02-09 16:52:11.154714936 +0000 UTC m=+5.484947625, 105.809787ms after stability testing.go:149: cluster: 2025-02-09T16:52:11.154Z [INFO] stable state for Leader reached at 2025-02-09 16:52:11.04890515 +0000 UTC m=+5.379137838 (1 nodes), 145.443908ms from start of poll, 434.55482ms from cluster start. Timeout at 2025-02-09 16:52:11.154714936 +0000 UTC m=+5.484947625, 105.809787ms after stability raft_test.go:371: [INFO] Reconnecting Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Candidate] testLoggerAdapter verbose: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.165Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering follower state: follower="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Follower]" leader=77661ac0-09bb-b789-ba94-ff98cbd2fafc testing.go:149: server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: 2025-02-09T16:52:11.165Z [INFO] server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a: entering follower state: follower="Node at c7bfc503-0499-6397-f90c-b7fe4bb4db0a [Follower]" leader=77661ac0-09bb-b789-ba94-ff98cbd2fafc testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.166Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: pipelining replication: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.166Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: pipelining replication: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.171Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: aborting pipeline replication: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.171Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: aborting pipeline replication: peer="{Voter server-c7bfc503-0499-6397-f90c-b7fe4bb4db0a c7bfc503-0499-6397-f90c-b7fe4bb4db0a}" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.171Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: aborting pipeline replication: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.171Z [INFO] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: aborting pipeline replication: peer="{Voter server-ed3df739-60f3-04f4-cdd3-6aa793b08087 ed3df739-60f3-04f4-cdd3-6aa793b08087}" testLoggerAdapter verbose: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.221Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to heartbeat to: peer=c7bfc503-0499-6397-f90c-b7fe4bb4db0a error="command timed out" testing.go:149: server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: 2025-02-09T16:52:11.221Z [ERROR] server-77661ac0-09bb-b789-ba94-ff98cbd2fafc: failed to heartbeat to: peer=c7bfc503-0499-6397-f90c-b7fe4bb4db0a error="command timed out" --- PASS: TestRaft_LeaderFail (0.61s) === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.222Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06ffd059-69aa-e354-e84c-f264a1c5f4d8 Address:06ffd059-69aa-e354-e84c-f264a1c5f4d8} {Suffrage:Voter ID:server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 Address:0936aadc-eb10-aaff-a82c-3a2bce0bf5d6} {Suffrage:Voter ID:server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 Address:77f9f4ee-cb0b-fe10-e2a5-aa304da7d795}]" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.222Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06ffd059-69aa-e354-e84c-f264a1c5f4d8 Address:06ffd059-69aa-e354-e84c-f264a1c5f4d8} {Suffrage:Voter ID:server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 Address:0936aadc-eb10-aaff-a82c-3a2bce0bf5d6} {Suffrage:Voter ID:server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 Address:77f9f4ee-cb0b-fe10-e2a5-aa304da7d795}]" testLoggerAdapter verbose: server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6: 2025-02-09T16:52:11.222Z [INFO] server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06ffd059-69aa-e354-e84c-f264a1c5f4d8 Address:06ffd059-69aa-e354-e84c-f264a1c5f4d8} {Suffrage:Voter ID:server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 Address:0936aadc-eb10-aaff-a82c-3a2bce0bf5d6} {Suffrage:Voter ID:server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 Address:77f9f4ee-cb0b-fe10-e2a5-aa304da7d795}]" testing.go:149: server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6: 2025-02-09T16:52:11.222Z [INFO] server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06ffd059-69aa-e354-e84c-f264a1c5f4d8 Address:06ffd059-69aa-e354-e84c-f264a1c5f4d8} {Suffrage:Voter ID:server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 Address:0936aadc-eb10-aaff-a82c-3a2bce0bf5d6} {Suffrage:Voter ID:server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 Address:77f9f4ee-cb0b-fe10-e2a5-aa304da7d795}]" testLoggerAdapter verbose: server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795: 2025-02-09T16:52:11.223Z [INFO] server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06ffd059-69aa-e354-e84c-f264a1c5f4d8 Address:06ffd059-69aa-e354-e84c-f264a1c5f4d8} {Suffrage:Voter ID:server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 Address:0936aadc-eb10-aaff-a82c-3a2bce0bf5d6} {Suffrage:Voter ID:server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 Address:77f9f4ee-cb0b-fe10-e2a5-aa304da7d795}]" testing.go:149: server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795: 2025-02-09T16:52:11.223Z [INFO] server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06ffd059-69aa-e354-e84c-f264a1c5f4d8 Address:06ffd059-69aa-e354-e84c-f264a1c5f4d8} {Suffrage:Voter ID:server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 Address:0936aadc-eb10-aaff-a82c-3a2bce0bf5d6} {Suffrage:Voter ID:server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 Address:77f9f4ee-cb0b-fe10-e2a5-aa304da7d795}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.223Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:11.223Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.223Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: entering follower state: follower="Node at 06ffd059-69aa-e354-e84c-f264a1c5f4d8 [Follower]" leader= testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.223Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: entering follower state: follower="Node at 06ffd059-69aa-e354-e84c-f264a1c5f4d8 [Follower]" leader= testLoggerAdapter verbose: server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6: 2025-02-09T16:52:11.223Z [INFO] server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6: entering follower state: follower="Node at 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 [Follower]" leader= testing.go:149: server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6: 2025-02-09T16:52:11.223Z [INFO] server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6: entering follower state: follower="Node at 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 [Follower]" leader= testLoggerAdapter verbose: server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795: 2025-02-09T16:52:11.223Z [INFO] server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795: entering follower state: follower="Node at 77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 [Follower]" leader= testing.go:149: server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795: 2025-02-09T16:52:11.223Z [INFO] server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795: entering follower state: follower="Node at 77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 [Follower]" leader= testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [WARN] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [WARN] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: entering candidate state: node="Node at 06ffd059-69aa-e354-e84c-f264a1c5f4d8 [Candidate]" term=2 testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: entering candidate state: node="Node at 06ffd059-69aa-e354-e84c-f264a1c5f4d8 [Candidate]" term=2 testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: election won: tally=2 testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: election won: tally=2 testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: entering leader state: leader="Node at 06ffd059-69aa-e354-e84c-f264a1c5f4d8 [Leader]" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: entering leader state: leader="Node at 06ffd059-69aa-e354-e84c-f264a1c5f4d8 [Leader]" testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: added peer, starting replication: peer=server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: added peer, starting replication: peer=server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: added peer, starting replication: peer=server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: added peer, starting replication: peer=server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: pipelining replication: peer="{Voter server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 77f9f4ee-cb0b-fe10-e2a5-aa304da7d795}" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: pipelining replication: peer="{Voter server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 77f9f4ee-cb0b-fe10-e2a5-aa304da7d795}" testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: pipelining replication: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.290Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: pipelining replication: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.396Z [INFO] stable state for Leader reached at 2025-02-09 16:52:11.29101062 +0000 UTC m=+5.621243316 (1 nodes), 67.911602ms from start of poll, 68.344374ms from cluster start. Timeout at 2025-02-09 16:52:11.396634524 +0000 UTC m=+5.726867210, 105.623894ms after stability testing.go:149: cluster: 2025-02-09T16:52:11.396Z [INFO] stable state for Leader reached at 2025-02-09 16:52:11.29101062 +0000 UTC m=+5.621243316 (1 nodes), 67.911602ms from start of poll, 68.344374ms from cluster start. Timeout at 2025-02-09 16:52:11.396634524 +0000 UTC m=+5.726867210, 105.623894ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.396Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:11.396Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.502Z [INFO] stable state for Follower reached at 2025-02-09 16:52:11.3967714 +0000 UTC m=+5.727004088 (2 nodes), 970ns from start of poll, 174.105146ms from cluster start. Timeout at 2025-02-09 16:52:11.502442277 +0000 UTC m=+5.832674965, 105.670877ms after stability testing.go:149: cluster: 2025-02-09T16:52:11.502Z [INFO] stable state for Follower reached at 2025-02-09 16:52:11.3967714 +0000 UTC m=+5.727004088 (2 nodes), 970ns from start of poll, 174.105146ms from cluster start. Timeout at 2025-02-09 16:52:11.502442277 +0000 UTC m=+5.832674965, 105.670877ms after stability testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.502Z [ERROR] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: failed to pipeline appendEntries: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" error="append pipeline closed" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.502Z [ERROR] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: failed to pipeline appendEntries: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" error="append pipeline closed" testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.502Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: aborting pipeline replication: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.502Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: aborting pipeline replication: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.502Z [ERROR] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: failed to appendEntries to: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" error="failed to connect to peer: 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.502Z [ERROR] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: failed to appendEntries to: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" error="failed to connect to peer: 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6" testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.503Z [ERROR] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: failed to heartbeat to: peer=0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 error="failed to connect to peer: 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.503Z [ERROR] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: failed to heartbeat to: peer=0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 error="failed to connect to peer: 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6" raft_test.go:420: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.513Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: pipelining replication: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.513Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: pipelining replication: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.526Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:11.526Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.632Z [INFO] stable state for Leader reached at 2025-02-09 16:52:11.526550674 +0000 UTC m=+5.856783360 (1 nodes), 1.204µs from start of poll, 303.884418ms from cluster start. Timeout at 2025-02-09 16:52:11.632174705 +0000 UTC m=+5.962407398, 105.624038ms after stability testing.go:149: cluster: 2025-02-09T16:52:11.632Z [INFO] stable state for Leader reached at 2025-02-09 16:52:11.526550674 +0000 UTC m=+5.856783360 (1 nodes), 1.204µs from start of poll, 303.884418ms from cluster start. Timeout at 2025-02-09 16:52:11.632174705 +0000 UTC m=+5.962407398, 105.624038ms after stability testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.632Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: aborting pipeline replication: peer="{Voter server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 77f9f4ee-cb0b-fe10-e2a5-aa304da7d795}" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.632Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: aborting pipeline replication: peer="{Voter server-77f9f4ee-cb0b-fe10-e2a5-aa304da7d795 77f9f4ee-cb0b-fe10-e2a5-aa304da7d795}" testLoggerAdapter verbose: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.632Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: aborting pipeline replication: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" testing.go:149: server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: 2025-02-09T16:52:11.632Z [INFO] server-06ffd059-69aa-e354-e84c-f264a1c5f4d8: aborting pipeline replication: peer="{Voter server-0936aadc-eb10-aaff-a82c-3a2bce0bf5d6 0936aadc-eb10-aaff-a82c-3a2bce0bf5d6}" --- PASS: TestRaft_BehindFollower (0.41s) === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.633Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d5e8375-71bb-338d-d0d6-0d8c4e571953 Address:2d5e8375-71bb-338d-d0d6-0d8c4e571953} {Suffrage:Voter ID:server-65d53bb2-3b27-bd36-2a22-df5356e7903c Address:65d53bb2-3b27-bd36-2a22-df5356e7903c} {Suffrage:Voter ID:server-c15fe453-254e-c06d-67f8-752fc90a3b1b Address:c15fe453-254e-c06d-67f8-752fc90a3b1b}]" testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.633Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d5e8375-71bb-338d-d0d6-0d8c4e571953 Address:2d5e8375-71bb-338d-d0d6-0d8c4e571953} {Suffrage:Voter ID:server-65d53bb2-3b27-bd36-2a22-df5356e7903c Address:65d53bb2-3b27-bd36-2a22-df5356e7903c} {Suffrage:Voter ID:server-c15fe453-254e-c06d-67f8-752fc90a3b1b Address:c15fe453-254e-c06d-67f8-752fc90a3b1b}]" testLoggerAdapter verbose: server-65d53bb2-3b27-bd36-2a22-df5356e7903c: 2025-02-09T16:52:11.633Z [INFO] server-65d53bb2-3b27-bd36-2a22-df5356e7903c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d5e8375-71bb-338d-d0d6-0d8c4e571953 Address:2d5e8375-71bb-338d-d0d6-0d8c4e571953} {Suffrage:Voter ID:server-65d53bb2-3b27-bd36-2a22-df5356e7903c Address:65d53bb2-3b27-bd36-2a22-df5356e7903c} {Suffrage:Voter ID:server-c15fe453-254e-c06d-67f8-752fc90a3b1b Address:c15fe453-254e-c06d-67f8-752fc90a3b1b}]" testing.go:149: server-65d53bb2-3b27-bd36-2a22-df5356e7903c: 2025-02-09T16:52:11.633Z [INFO] server-65d53bb2-3b27-bd36-2a22-df5356e7903c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d5e8375-71bb-338d-d0d6-0d8c4e571953 Address:2d5e8375-71bb-338d-d0d6-0d8c4e571953} {Suffrage:Voter ID:server-65d53bb2-3b27-bd36-2a22-df5356e7903c Address:65d53bb2-3b27-bd36-2a22-df5356e7903c} {Suffrage:Voter ID:server-c15fe453-254e-c06d-67f8-752fc90a3b1b Address:c15fe453-254e-c06d-67f8-752fc90a3b1b}]" testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.633Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: entering follower state: follower="Node at 2d5e8375-71bb-338d-d0d6-0d8c4e571953 [Follower]" leader= testLoggerAdapter verbose: server-c15fe453-254e-c06d-67f8-752fc90a3b1b: 2025-02-09T16:52:11.633Z [INFO] server-c15fe453-254e-c06d-67f8-752fc90a3b1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d5e8375-71bb-338d-d0d6-0d8c4e571953 Address:2d5e8375-71bb-338d-d0d6-0d8c4e571953} {Suffrage:Voter ID:server-65d53bb2-3b27-bd36-2a22-df5356e7903c Address:65d53bb2-3b27-bd36-2a22-df5356e7903c} {Suffrage:Voter ID:server-c15fe453-254e-c06d-67f8-752fc90a3b1b Address:c15fe453-254e-c06d-67f8-752fc90a3b1b}]" testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.633Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: entering follower state: follower="Node at 2d5e8375-71bb-338d-d0d6-0d8c4e571953 [Follower]" leader= testing.go:149: server-c15fe453-254e-c06d-67f8-752fc90a3b1b: 2025-02-09T16:52:11.633Z [INFO] server-c15fe453-254e-c06d-67f8-752fc90a3b1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d5e8375-71bb-338d-d0d6-0d8c4e571953 Address:2d5e8375-71bb-338d-d0d6-0d8c4e571953} {Suffrage:Voter ID:server-65d53bb2-3b27-bd36-2a22-df5356e7903c Address:65d53bb2-3b27-bd36-2a22-df5356e7903c} {Suffrage:Voter ID:server-c15fe453-254e-c06d-67f8-752fc90a3b1b Address:c15fe453-254e-c06d-67f8-752fc90a3b1b}]" testLoggerAdapter verbose: server-65d53bb2-3b27-bd36-2a22-df5356e7903c: 2025-02-09T16:52:11.633Z [INFO] server-65d53bb2-3b27-bd36-2a22-df5356e7903c: entering follower state: follower="Node at 65d53bb2-3b27-bd36-2a22-df5356e7903c [Follower]" leader= testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.633Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-65d53bb2-3b27-bd36-2a22-df5356e7903c: 2025-02-09T16:52:11.633Z [INFO] server-65d53bb2-3b27-bd36-2a22-df5356e7903c: entering follower state: follower="Node at 65d53bb2-3b27-bd36-2a22-df5356e7903c [Follower]" leader= testing.go:149: cluster: 2025-02-09T16:52:11.633Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c15fe453-254e-c06d-67f8-752fc90a3b1b: 2025-02-09T16:52:11.633Z [INFO] server-c15fe453-254e-c06d-67f8-752fc90a3b1b: entering follower state: follower="Node at c15fe453-254e-c06d-67f8-752fc90a3b1b [Follower]" leader= testing.go:149: server-c15fe453-254e-c06d-67f8-752fc90a3b1b: 2025-02-09T16:52:11.633Z [INFO] server-c15fe453-254e-c06d-67f8-752fc90a3b1b: entering follower state: follower="Node at c15fe453-254e-c06d-67f8-752fc90a3b1b [Follower]" leader= testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.696Z [WARN] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.696Z [WARN] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.696Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: entering candidate state: node="Node at 2d5e8375-71bb-338d-d0d6-0d8c4e571953 [Candidate]" term=2 testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.696Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: entering candidate state: node="Node at 2d5e8375-71bb-338d-d0d6-0d8c4e571953 [Candidate]" term=2 testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.696Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: election won: tally=2 testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.696Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: election won: tally=2 testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.697Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: entering leader state: leader="Node at 2d5e8375-71bb-338d-d0d6-0d8c4e571953 [Leader]" testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.697Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: entering leader state: leader="Node at 2d5e8375-71bb-338d-d0d6-0d8c4e571953 [Leader]" testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.697Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: added peer, starting replication: peer=server-65d53bb2-3b27-bd36-2a22-df5356e7903c testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.697Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: added peer, starting replication: peer=server-65d53bb2-3b27-bd36-2a22-df5356e7903c testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.697Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: added peer, starting replication: peer=server-c15fe453-254e-c06d-67f8-752fc90a3b1b testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.697Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: added peer, starting replication: peer=server-c15fe453-254e-c06d-67f8-752fc90a3b1b testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.698Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: pipelining replication: peer="{Voter server-c15fe453-254e-c06d-67f8-752fc90a3b1b c15fe453-254e-c06d-67f8-752fc90a3b1b}" testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.698Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: pipelining replication: peer="{Voter server-c15fe453-254e-c06d-67f8-752fc90a3b1b c15fe453-254e-c06d-67f8-752fc90a3b1b}" testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.698Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: pipelining replication: peer="{Voter server-65d53bb2-3b27-bd36-2a22-df5356e7903c 65d53bb2-3b27-bd36-2a22-df5356e7903c}" testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.698Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: pipelining replication: peer="{Voter server-65d53bb2-3b27-bd36-2a22-df5356e7903c 65d53bb2-3b27-bd36-2a22-df5356e7903c}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.803Z [INFO] stable state for Leader reached at 2025-02-09 16:52:11.698141449 +0000 UTC m=+6.028374139 (1 nodes), 64.642216ms from start of poll, 65.104733ms from cluster start. Timeout at 2025-02-09 16:52:11.803796472 +0000 UTC m=+6.134029160, 105.655021ms after stability testing.go:149: cluster: 2025-02-09T16:52:11.803Z [INFO] stable state for Leader reached at 2025-02-09 16:52:11.698141449 +0000 UTC m=+6.028374139 (1 nodes), 64.642216ms from start of poll, 65.104733ms from cluster start. Timeout at 2025-02-09 16:52:11.803796472 +0000 UTC m=+6.134029160, 105.655021ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.803Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:11.803Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.909Z [INFO] stable state for Follower reached at 2025-02-09 16:52:11.803940528 +0000 UTC m=+6.134173215 (2 nodes), 2.04µs from start of poll, 170.903809ms from cluster start. Timeout at 2025-02-09 16:52:11.909885156 +0000 UTC m=+6.240117844, 105.944629ms after stability testing.go:149: cluster: 2025-02-09T16:52:11.909Z [INFO] stable state for Follower reached at 2025-02-09 16:52:11.803940528 +0000 UTC m=+6.134173215 (2 nodes), 2.04µs from start of poll, 170.903809ms from cluster start. Timeout at 2025-02-09 16:52:11.909885156 +0000 UTC m=+6.240117844, 105.944629ms after stability testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.910Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: aborting pipeline replication: peer="{Voter server-c15fe453-254e-c06d-67f8-752fc90a3b1b c15fe453-254e-c06d-67f8-752fc90a3b1b}" testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.910Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: aborting pipeline replication: peer="{Voter server-c15fe453-254e-c06d-67f8-752fc90a3b1b c15fe453-254e-c06d-67f8-752fc90a3b1b}" testLoggerAdapter verbose: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.910Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: aborting pipeline replication: peer="{Voter server-65d53bb2-3b27-bd36-2a22-df5356e7903c 65d53bb2-3b27-bd36-2a22-df5356e7903c}" testing.go:149: server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: 2025-02-09T16:52:11.910Z [INFO] server-2d5e8375-71bb-338d-d0d6-0d8c4e571953: aborting pipeline replication: peer="{Voter server-65d53bb2-3b27-bd36-2a22-df5356e7903c 65d53bb2-3b27-bd36-2a22-df5356e7903c}" --- PASS: TestRaft_ApplyNonLeader (0.28s) === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e: 2025-02-09T16:52:11.910Z [INFO] server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e Address:31553d2c-8960-b9c9-0e16-ef8f8be4c95e} {Suffrage:Voter ID:server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d Address:798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d} {Suffrage:Voter ID:server-8c62e9ff-2000-1e06-839f-4e47f980fde0 Address:8c62e9ff-2000-1e06-839f-4e47f980fde0}]" testing.go:149: server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e: 2025-02-09T16:52:11.910Z [INFO] server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e Address:31553d2c-8960-b9c9-0e16-ef8f8be4c95e} {Suffrage:Voter ID:server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d Address:798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d} {Suffrage:Voter ID:server-8c62e9ff-2000-1e06-839f-4e47f980fde0 Address:8c62e9ff-2000-1e06-839f-4e47f980fde0}]" testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:11.910Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e Address:31553d2c-8960-b9c9-0e16-ef8f8be4c95e} {Suffrage:Voter ID:server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d Address:798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d} {Suffrage:Voter ID:server-8c62e9ff-2000-1e06-839f-4e47f980fde0 Address:8c62e9ff-2000-1e06-839f-4e47f980fde0}]" testLoggerAdapter verbose: server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e: 2025-02-09T16:52:11.910Z [INFO] server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e: entering follower state: follower="Node at 31553d2c-8960-b9c9-0e16-ef8f8be4c95e [Follower]" leader= testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:11.910Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e Address:31553d2c-8960-b9c9-0e16-ef8f8be4c95e} {Suffrage:Voter ID:server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d Address:798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d} {Suffrage:Voter ID:server-8c62e9ff-2000-1e06-839f-4e47f980fde0 Address:8c62e9ff-2000-1e06-839f-4e47f980fde0}]" testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:11.910Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: entering follower state: follower="Node at 798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d [Follower]" leader= testing.go:149: server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e: 2025-02-09T16:52:11.910Z [INFO] server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e: entering follower state: follower="Node at 31553d2c-8960-b9c9-0e16-ef8f8be4c95e [Follower]" leader= testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:11.910Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: entering follower state: follower="Node at 798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d [Follower]" leader= testLoggerAdapter verbose: server-8c62e9ff-2000-1e06-839f-4e47f980fde0: 2025-02-09T16:52:11.911Z [INFO] server-8c62e9ff-2000-1e06-839f-4e47f980fde0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e Address:31553d2c-8960-b9c9-0e16-ef8f8be4c95e} {Suffrage:Voter ID:server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d Address:798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d} {Suffrage:Voter ID:server-8c62e9ff-2000-1e06-839f-4e47f980fde0 Address:8c62e9ff-2000-1e06-839f-4e47f980fde0}]" testing.go:149: server-8c62e9ff-2000-1e06-839f-4e47f980fde0: 2025-02-09T16:52:11.911Z [INFO] server-8c62e9ff-2000-1e06-839f-4e47f980fde0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e Address:31553d2c-8960-b9c9-0e16-ef8f8be4c95e} {Suffrage:Voter ID:server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d Address:798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d} {Suffrage:Voter ID:server-8c62e9ff-2000-1e06-839f-4e47f980fde0 Address:8c62e9ff-2000-1e06-839f-4e47f980fde0}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:11.911Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:11.911Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8c62e9ff-2000-1e06-839f-4e47f980fde0: 2025-02-09T16:52:11.911Z [INFO] server-8c62e9ff-2000-1e06-839f-4e47f980fde0: entering follower state: follower="Node at 8c62e9ff-2000-1e06-839f-4e47f980fde0 [Follower]" leader= testing.go:149: server-8c62e9ff-2000-1e06-839f-4e47f980fde0: 2025-02-09T16:52:11.911Z [INFO] server-8c62e9ff-2000-1e06-839f-4e47f980fde0: entering follower state: follower="Node at 8c62e9ff-2000-1e06-839f-4e47f980fde0 [Follower]" leader= testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [WARN] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [WARN] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: entering candidate state: node="Node at 798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d [Candidate]" term=2 testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: entering candidate state: node="Node at 798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d [Candidate]" term=2 testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: election won: tally=2 testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: election won: tally=2 testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: entering leader state: leader="Node at 798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d [Leader]" testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: entering leader state: leader="Node at 798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d [Leader]" testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: added peer, starting replication: peer=server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: added peer, starting replication: peer=server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: added peer, starting replication: peer=server-8c62e9ff-2000-1e06-839f-4e47f980fde0 testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: added peer, starting replication: peer=server-8c62e9ff-2000-1e06-839f-4e47f980fde0 testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: pipelining replication: peer="{Voter server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e 31553d2c-8960-b9c9-0e16-ef8f8be4c95e}" testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: pipelining replication: peer="{Voter server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e 31553d2c-8960-b9c9-0e16-ef8f8be4c95e}" testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: pipelining replication: peer="{Voter server-8c62e9ff-2000-1e06-839f-4e47f980fde0 8c62e9ff-2000-1e06-839f-4e47f980fde0}" testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.024Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: pipelining replication: peer="{Voter server-8c62e9ff-2000-1e06-839f-4e47f980fde0 8c62e9ff-2000-1e06-839f-4e47f980fde0}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:12.230Z [INFO] stable state for Leader reached at 2025-02-09 16:52:12.024787828 +0000 UTC m=+6.355020524 (1 nodes), 113.567041ms from start of poll, 114.086659ms from cluster start. Timeout at 2025-02-09 16:52:12.230095105 +0000 UTC m=+6.560327793, 205.307269ms after stability testing.go:149: cluster: 2025-02-09T16:52:12.230Z [INFO] stable state for Leader reached at 2025-02-09 16:52:12.024787828 +0000 UTC m=+6.355020524 (1 nodes), 113.567041ms from start of poll, 114.086659ms from cluster start. Timeout at 2025-02-09 16:52:12.230095105 +0000 UTC m=+6.560327793, 205.307269ms after stability testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.243Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: aborting pipeline replication: peer="{Voter server-8c62e9ff-2000-1e06-839f-4e47f980fde0 8c62e9ff-2000-1e06-839f-4e47f980fde0}" testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.243Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: aborting pipeline replication: peer="{Voter server-8c62e9ff-2000-1e06-839f-4e47f980fde0 8c62e9ff-2000-1e06-839f-4e47f980fde0}" testLoggerAdapter verbose: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.243Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: aborting pipeline replication: peer="{Voter server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e 31553d2c-8960-b9c9-0e16-ef8f8be4c95e}" testing.go:149: server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: 2025-02-09T16:52:12.243Z [INFO] server-798b00cf-1457-f24e-e0ce-f0f8fc3e9d3d: aborting pipeline replication: peer="{Voter server-31553d2c-8960-b9c9-0e16-ef8f8be4c95e 31553d2c-8960-b9c9-0e16-ef8f8be4c95e}" --- PASS: TestRaft_ApplyConcurrent (0.33s) === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.243Z [INFO] server-45951805-526c-c287-56e1-44c246e01f0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45951805-526c-c287-56e1-44c246e01f0a Address:45951805-526c-c287-56e1-44c246e01f0a}]" testing.go:149: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.243Z [INFO] server-45951805-526c-c287-56e1-44c246e01f0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45951805-526c-c287-56e1-44c246e01f0a Address:45951805-526c-c287-56e1-44c246e01f0a}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:12.243Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:12.243Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.244Z [INFO] server-45951805-526c-c287-56e1-44c246e01f0a: entering follower state: follower="Node at 45951805-526c-c287-56e1-44c246e01f0a [Follower]" leader= testing.go:149: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.244Z [INFO] server-45951805-526c-c287-56e1-44c246e01f0a: entering follower state: follower="Node at 45951805-526c-c287-56e1-44c246e01f0a [Follower]" leader= testLoggerAdapter verbose: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.357Z [WARN] server-45951805-526c-c287-56e1-44c246e01f0a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.357Z [WARN] server-45951805-526c-c287-56e1-44c246e01f0a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.357Z [INFO] server-45951805-526c-c287-56e1-44c246e01f0a: entering candidate state: node="Node at 45951805-526c-c287-56e1-44c246e01f0a [Candidate]" term=2 testing.go:149: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.357Z [INFO] server-45951805-526c-c287-56e1-44c246e01f0a: entering candidate state: node="Node at 45951805-526c-c287-56e1-44c246e01f0a [Candidate]" term=2 testLoggerAdapter verbose: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.357Z [INFO] server-45951805-526c-c287-56e1-44c246e01f0a: election won: tally=1 testing.go:149: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.357Z [INFO] server-45951805-526c-c287-56e1-44c246e01f0a: election won: tally=1 testLoggerAdapter verbose: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.357Z [INFO] server-45951805-526c-c287-56e1-44c246e01f0a: entering leader state: leader="Node at 45951805-526c-c287-56e1-44c246e01f0a [Leader]" testing.go:149: server-45951805-526c-c287-56e1-44c246e01f0a: 2025-02-09T16:52:12.357Z [INFO] server-45951805-526c-c287-56e1-44c246e01f0a: entering leader state: leader="Node at 45951805-526c-c287-56e1-44c246e01f0a [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:12.558Z [INFO] stable state for Leader reached at 2025-02-09 16:52:12.357633314 +0000 UTC m=+6.687866004 (1 nodes), 113.624345ms from start of poll, 113.827876ms from cluster start. Timeout at 2025-02-09 16:52:12.558857083 +0000 UTC m=+6.889089774, 201.22377ms after stability testing.go:149: cluster: 2025-02-09T16:52:12.558Z [INFO] stable state for Leader reached at 2025-02-09 16:52:12.357633314 +0000 UTC m=+6.687866004 (1 nodes), 113.624345ms from start of poll, 113.827876ms from cluster start. Timeout at 2025-02-09 16:52:12.558857083 +0000 UTC m=+6.889089774, 201.22377ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.32s) === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-ec146fd5-f24a-ea53-7987-157c5579fbd6: 2025-02-09T16:52:12.560Z [INFO] server-ec146fd5-f24a-ea53-7987-157c5579fbd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec146fd5-f24a-ea53-7987-157c5579fbd6 Address:ec146fd5-f24a-ea53-7987-157c5579fbd6} {Suffrage:Voter ID:server-7a479977-a20d-2fb7-51b1-b45fe704bfd7 Address:7a479977-a20d-2fb7-51b1-b45fe704bfd7}]" testing.go:149: server-ec146fd5-f24a-ea53-7987-157c5579fbd6: 2025-02-09T16:52:12.560Z [INFO] server-ec146fd5-f24a-ea53-7987-157c5579fbd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec146fd5-f24a-ea53-7987-157c5579fbd6 Address:ec146fd5-f24a-ea53-7987-157c5579fbd6} {Suffrage:Voter ID:server-7a479977-a20d-2fb7-51b1-b45fe704bfd7 Address:7a479977-a20d-2fb7-51b1-b45fe704bfd7}]" testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.561Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec146fd5-f24a-ea53-7987-157c5579fbd6 Address:ec146fd5-f24a-ea53-7987-157c5579fbd6} {Suffrage:Voter ID:server-7a479977-a20d-2fb7-51b1-b45fe704bfd7 Address:7a479977-a20d-2fb7-51b1-b45fe704bfd7}]" testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.561Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec146fd5-f24a-ea53-7987-157c5579fbd6 Address:ec146fd5-f24a-ea53-7987-157c5579fbd6} {Suffrage:Voter ID:server-7a479977-a20d-2fb7-51b1-b45fe704bfd7 Address:7a479977-a20d-2fb7-51b1-b45fe704bfd7}]" testLoggerAdapter verbose: server-ce866238-52b5-8e62-c1cf-dc3a7b284911: 2025-02-09T16:52:12.561Z [INFO] server-ce866238-52b5-8e62-c1cf-dc3a7b284911: initial configuration: index=0 servers=[] testing.go:149: server-ce866238-52b5-8e62-c1cf-dc3a7b284911: 2025-02-09T16:52:12.561Z [INFO] server-ce866238-52b5-8e62-c1cf-dc3a7b284911: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2025-02-09T16:52:12.561Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:12.561Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ec146fd5-f24a-ea53-7987-157c5579fbd6: 2025-02-09T16:52:12.561Z [INFO] server-ec146fd5-f24a-ea53-7987-157c5579fbd6: entering follower state: follower="Node at ec146fd5-f24a-ea53-7987-157c5579fbd6 [Follower]" leader= testing.go:149: server-ec146fd5-f24a-ea53-7987-157c5579fbd6: 2025-02-09T16:52:12.561Z [INFO] server-ec146fd5-f24a-ea53-7987-157c5579fbd6: entering follower state: follower="Node at ec146fd5-f24a-ea53-7987-157c5579fbd6 [Follower]" leader= testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.561Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: entering follower state: follower="Node at 7a479977-a20d-2fb7-51b1-b45fe704bfd7 [Follower]" leader= testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.561Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: entering follower state: follower="Node at 7a479977-a20d-2fb7-51b1-b45fe704bfd7 [Follower]" leader= testLoggerAdapter verbose: server-ce866238-52b5-8e62-c1cf-dc3a7b284911: 2025-02-09T16:52:12.561Z [INFO] server-ce866238-52b5-8e62-c1cf-dc3a7b284911: entering follower state: follower="Node at ce866238-52b5-8e62-c1cf-dc3a7b284911 [Follower]" leader= testing.go:149: server-ce866238-52b5-8e62-c1cf-dc3a7b284911: 2025-02-09T16:52:12.561Z [INFO] server-ce866238-52b5-8e62-c1cf-dc3a7b284911: entering follower state: follower="Node at ce866238-52b5-8e62-c1cf-dc3a7b284911 [Follower]" leader= testLoggerAdapter verbose: server-ce866238-52b5-8e62-c1cf-dc3a7b284911: 2025-02-09T16:52:12.615Z [WARN] server-ce866238-52b5-8e62-c1cf-dc3a7b284911: no known peers, aborting election testing.go:149: server-ce866238-52b5-8e62-c1cf-dc3a7b284911: 2025-02-09T16:52:12.615Z [WARN] server-ce866238-52b5-8e62-c1cf-dc3a7b284911: no known peers, aborting election testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.626Z [WARN] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.626Z [WARN] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.627Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: entering candidate state: node="Node at 7a479977-a20d-2fb7-51b1-b45fe704bfd7 [Candidate]" term=2 testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.627Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: entering candidate state: node="Node at 7a479977-a20d-2fb7-51b1-b45fe704bfd7 [Candidate]" term=2 testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.628Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: election won: tally=2 testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.628Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: election won: tally=2 testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.628Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: entering leader state: leader="Node at 7a479977-a20d-2fb7-51b1-b45fe704bfd7 [Leader]" testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.628Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: entering leader state: leader="Node at 7a479977-a20d-2fb7-51b1-b45fe704bfd7 [Leader]" testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.628Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: added peer, starting replication: peer=server-ec146fd5-f24a-ea53-7987-157c5579fbd6 testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.628Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: added peer, starting replication: peer=server-ec146fd5-f24a-ea53-7987-157c5579fbd6 testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.628Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: pipelining replication: peer="{Voter server-ec146fd5-f24a-ea53-7987-157c5579fbd6 ec146fd5-f24a-ea53-7987-157c5579fbd6}" testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.628Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: pipelining replication: peer="{Voter server-ec146fd5-f24a-ea53-7987-157c5579fbd6 ec146fd5-f24a-ea53-7987-157c5579fbd6}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:12.734Z [INFO] stable state for Leader reached at 2025-02-09 16:52:12.628938631 +0000 UTC m=+6.959171326 (1 nodes), 67.613522ms from start of poll, 68.112593ms from cluster start. Timeout at 2025-02-09 16:52:12.734876769 +0000 UTC m=+7.065109462, 105.938136ms after stability testing.go:149: cluster: 2025-02-09T16:52:12.734Z [INFO] stable state for Leader reached at 2025-02-09 16:52:12.628938631 +0000 UTC m=+6.959171326 (1 nodes), 67.613522ms from start of poll, 68.112593ms from cluster start. Timeout at 2025-02-09 16:52:12.734876769 +0000 UTC m=+7.065109462, 105.938136ms after stability testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.735Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: updating configuration: command=AddStaging server-id=server-ce866238-52b5-8e62-c1cf-dc3a7b284911 server-addr=ce866238-52b5-8e62-c1cf-dc3a7b284911 servers="[{Suffrage:Voter ID:server-ec146fd5-f24a-ea53-7987-157c5579fbd6 Address:ec146fd5-f24a-ea53-7987-157c5579fbd6} {Suffrage:Voter ID:server-7a479977-a20d-2fb7-51b1-b45fe704bfd7 Address:7a479977-a20d-2fb7-51b1-b45fe704bfd7} {Suffrage:Voter ID:server-ce866238-52b5-8e62-c1cf-dc3a7b284911 Address:ce866238-52b5-8e62-c1cf-dc3a7b284911}]" testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.735Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: updating configuration: command=AddStaging server-id=server-ce866238-52b5-8e62-c1cf-dc3a7b284911 server-addr=ce866238-52b5-8e62-c1cf-dc3a7b284911 servers="[{Suffrage:Voter ID:server-ec146fd5-f24a-ea53-7987-157c5579fbd6 Address:ec146fd5-f24a-ea53-7987-157c5579fbd6} {Suffrage:Voter ID:server-7a479977-a20d-2fb7-51b1-b45fe704bfd7 Address:7a479977-a20d-2fb7-51b1-b45fe704bfd7} {Suffrage:Voter ID:server-ce866238-52b5-8e62-c1cf-dc3a7b284911 Address:ce866238-52b5-8e62-c1cf-dc3a7b284911}]" testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.735Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: added peer, starting replication: peer=server-ce866238-52b5-8e62-c1cf-dc3a7b284911 testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.735Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: added peer, starting replication: peer=server-ce866238-52b5-8e62-c1cf-dc3a7b284911 testLoggerAdapter verbose: server-ce866238-52b5-8e62-c1cf-dc3a7b284911: 2025-02-09T16:52:12.735Z [WARN] server-ce866238-52b5-8e62-c1cf-dc3a7b284911: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-ce866238-52b5-8e62-c1cf-dc3a7b284911: 2025-02-09T16:52:12.735Z [WARN] server-ce866238-52b5-8e62-c1cf-dc3a7b284911: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.735Z [WARN] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: appendEntries rejected, sending older logs: peer="{Voter server-ce866238-52b5-8e62-c1cf-dc3a7b284911 ce866238-52b5-8e62-c1cf-dc3a7b284911}" next=1 testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.735Z [WARN] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: appendEntries rejected, sending older logs: peer="{Voter server-ce866238-52b5-8e62-c1cf-dc3a7b284911 ce866238-52b5-8e62-c1cf-dc3a7b284911}" next=1 testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.735Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: pipelining replication: peer="{Voter server-ce866238-52b5-8e62-c1cf-dc3a7b284911 ce866238-52b5-8e62-c1cf-dc3a7b284911}" testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.735Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: pipelining replication: peer="{Voter server-ce866238-52b5-8e62-c1cf-dc3a7b284911 ce866238-52b5-8e62-c1cf-dc3a7b284911}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:12.735Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:12.735Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:12.841Z [INFO] stable state for Leader reached at 2025-02-09 16:52:12.735517657 +0000 UTC m=+7.065750346 (1 nodes), 651ns from start of poll, 174.691613ms from cluster start. Timeout at 2025-02-09 16:52:12.841236795 +0000 UTC m=+7.171469486, 105.71914ms after stability testing.go:149: cluster: 2025-02-09T16:52:12.841Z [INFO] stable state for Leader reached at 2025-02-09 16:52:12.735517657 +0000 UTC m=+7.065750346 (1 nodes), 651ns from start of poll, 174.691613ms from cluster start. Timeout at 2025-02-09 16:52:12.841236795 +0000 UTC m=+7.171469486, 105.71914ms after stability testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.841Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: aborting pipeline replication: peer="{Voter server-ce866238-52b5-8e62-c1cf-dc3a7b284911 ce866238-52b5-8e62-c1cf-dc3a7b284911}" testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.841Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: aborting pipeline replication: peer="{Voter server-ce866238-52b5-8e62-c1cf-dc3a7b284911 ce866238-52b5-8e62-c1cf-dc3a7b284911}" testLoggerAdapter verbose: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.841Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: aborting pipeline replication: peer="{Voter server-ec146fd5-f24a-ea53-7987-157c5579fbd6 ec146fd5-f24a-ea53-7987-157c5579fbd6}" testing.go:149: server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: 2025-02-09T16:52:12.841Z [INFO] server-7a479977-a20d-2fb7-51b1-b45fe704bfd7: aborting pipeline replication: peer="{Voter server-ec146fd5-f24a-ea53-7987-157c5579fbd6 ec146fd5-f24a-ea53-7987-157c5579fbd6}" --- PASS: TestRaft_JoinNode (0.28s) === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.842Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02d96105-833a-caf2-58f7-620568a4aef9 Address:02d96105-833a-caf2-58f7-620568a4aef9}]" testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.842Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02d96105-833a-caf2-58f7-620568a4aef9 Address:02d96105-833a-caf2-58f7-620568a4aef9}]" testLoggerAdapter verbose: server-6b3ebfb3-99db-5bb7-8125-271801257201: 2025-02-09T16:52:12.842Z [INFO] server-6b3ebfb3-99db-5bb7-8125-271801257201: initial configuration: index=0 servers=[] testing.go:149: server-6b3ebfb3-99db-5bb7-8125-271801257201: 2025-02-09T16:52:12.842Z [INFO] server-6b3ebfb3-99db-5bb7-8125-271801257201: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-4786ac2c-2633-9a2e-292b-5891a4d4df85: 2025-02-09T16:52:12.842Z [INFO] server-4786ac2c-2633-9a2e-292b-5891a4d4df85: initial configuration: index=0 servers=[] testing.go:149: server-4786ac2c-2633-9a2e-292b-5891a4d4df85: 2025-02-09T16:52:12.842Z [INFO] server-4786ac2c-2633-9a2e-292b-5891a4d4df85: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2025-02-09T16:52:12.842Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:12.842Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.842Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: entering follower state: follower="Node at 02d96105-833a-caf2-58f7-620568a4aef9 [Follower]" leader= testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.842Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: entering follower state: follower="Node at 02d96105-833a-caf2-58f7-620568a4aef9 [Follower]" leader= testLoggerAdapter verbose: server-6b3ebfb3-99db-5bb7-8125-271801257201: 2025-02-09T16:52:12.842Z [INFO] server-6b3ebfb3-99db-5bb7-8125-271801257201: entering follower state: follower="Node at 6b3ebfb3-99db-5bb7-8125-271801257201 [Follower]" leader= testing.go:149: server-6b3ebfb3-99db-5bb7-8125-271801257201: 2025-02-09T16:52:12.842Z [INFO] server-6b3ebfb3-99db-5bb7-8125-271801257201: entering follower state: follower="Node at 6b3ebfb3-99db-5bb7-8125-271801257201 [Follower]" leader= testLoggerAdapter verbose: server-4786ac2c-2633-9a2e-292b-5891a4d4df85: 2025-02-09T16:52:12.842Z [INFO] server-4786ac2c-2633-9a2e-292b-5891a4d4df85: entering follower state: follower="Node at 4786ac2c-2633-9a2e-292b-5891a4d4df85 [Follower]" leader= testing.go:149: server-4786ac2c-2633-9a2e-292b-5891a4d4df85: 2025-02-09T16:52:12.842Z [INFO] server-4786ac2c-2633-9a2e-292b-5891a4d4df85: entering follower state: follower="Node at 4786ac2c-2633-9a2e-292b-5891a4d4df85 [Follower]" leader= testLoggerAdapter verbose: server-4786ac2c-2633-9a2e-292b-5891a4d4df85: 2025-02-09T16:52:12.922Z [WARN] server-4786ac2c-2633-9a2e-292b-5891a4d4df85: no known peers, aborting election testing.go:149: server-4786ac2c-2633-9a2e-292b-5891a4d4df85: 2025-02-09T16:52:12.922Z [WARN] server-4786ac2c-2633-9a2e-292b-5891a4d4df85: no known peers, aborting election testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.931Z [WARN] server-02d96105-833a-caf2-58f7-620568a4aef9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.931Z [WARN] server-02d96105-833a-caf2-58f7-620568a4aef9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.931Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: entering candidate state: node="Node at 02d96105-833a-caf2-58f7-620568a4aef9 [Candidate]" term=2 testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.931Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: entering candidate state: node="Node at 02d96105-833a-caf2-58f7-620568a4aef9 [Candidate]" term=2 testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.931Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: election won: tally=1 testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.931Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: election won: tally=1 testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.931Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: entering leader state: leader="Node at 02d96105-833a-caf2-58f7-620568a4aef9 [Leader]" testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:12.931Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: entering leader state: leader="Node at 02d96105-833a-caf2-58f7-620568a4aef9 [Leader]" testLoggerAdapter verbose: server-6b3ebfb3-99db-5bb7-8125-271801257201: 2025-02-09T16:52:12.943Z [WARN] server-6b3ebfb3-99db-5bb7-8125-271801257201: no known peers, aborting election testing.go:149: server-6b3ebfb3-99db-5bb7-8125-271801257201: 2025-02-09T16:52:12.943Z [WARN] server-6b3ebfb3-99db-5bb7-8125-271801257201: no known peers, aborting election testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.037Z [INFO] stable state for Leader reached at 2025-02-09 16:52:12.931863158 +0000 UTC m=+7.262095851 (1 nodes), 89.334239ms from start of poll, 89.842925ms from cluster start. Timeout at 2025-02-09 16:52:13.037856352 +0000 UTC m=+7.368089048, 105.993197ms after stability testing.go:149: cluster: 2025-02-09T16:52:13.037Z [INFO] stable state for Leader reached at 2025-02-09 16:52:12.931863158 +0000 UTC m=+7.262095851 (1 nodes), 89.334239ms from start of poll, 89.842925ms from cluster start. Timeout at 2025-02-09 16:52:13.037856352 +0000 UTC m=+7.368089048, 105.993197ms after stability testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.037Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: updating configuration: command=AddStaging server-id=server-6b3ebfb3-99db-5bb7-8125-271801257201 server-addr=6b3ebfb3-99db-5bb7-8125-271801257201 servers="[{Suffrage:Voter ID:server-02d96105-833a-caf2-58f7-620568a4aef9 Address:02d96105-833a-caf2-58f7-620568a4aef9} {Suffrage:Voter ID:server-6b3ebfb3-99db-5bb7-8125-271801257201 Address:6b3ebfb3-99db-5bb7-8125-271801257201}]" testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.037Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: updating configuration: command=AddStaging server-id=server-6b3ebfb3-99db-5bb7-8125-271801257201 server-addr=6b3ebfb3-99db-5bb7-8125-271801257201 servers="[{Suffrage:Voter ID:server-02d96105-833a-caf2-58f7-620568a4aef9 Address:02d96105-833a-caf2-58f7-620568a4aef9} {Suffrage:Voter ID:server-6b3ebfb3-99db-5bb7-8125-271801257201 Address:6b3ebfb3-99db-5bb7-8125-271801257201}]" testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.038Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: added peer, starting replication: peer=server-6b3ebfb3-99db-5bb7-8125-271801257201 testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.038Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: added peer, starting replication: peer=server-6b3ebfb3-99db-5bb7-8125-271801257201 testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.038Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:13.038Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6b3ebfb3-99db-5bb7-8125-271801257201: 2025-02-09T16:52:13.038Z [WARN] server-6b3ebfb3-99db-5bb7-8125-271801257201: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-6b3ebfb3-99db-5bb7-8125-271801257201: 2025-02-09T16:52:13.038Z [WARN] server-6b3ebfb3-99db-5bb7-8125-271801257201: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.038Z [WARN] server-02d96105-833a-caf2-58f7-620568a4aef9: appendEntries rejected, sending older logs: peer="{Voter server-6b3ebfb3-99db-5bb7-8125-271801257201 6b3ebfb3-99db-5bb7-8125-271801257201}" next=1 testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.038Z [WARN] server-02d96105-833a-caf2-58f7-620568a4aef9: appendEntries rejected, sending older logs: peer="{Voter server-6b3ebfb3-99db-5bb7-8125-271801257201 6b3ebfb3-99db-5bb7-8125-271801257201}" next=1 testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.038Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: pipelining replication: peer="{Voter server-6b3ebfb3-99db-5bb7-8125-271801257201 6b3ebfb3-99db-5bb7-8125-271801257201}" testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.038Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: pipelining replication: peer="{Voter server-6b3ebfb3-99db-5bb7-8125-271801257201 6b3ebfb3-99db-5bb7-8125-271801257201}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.143Z [INFO] stable state for Leader reached at 2025-02-09 16:52:13.03817071 +0000 UTC m=+7.368403396 (1 nodes), 677ns from start of poll, 196.15047ms from cluster start. Timeout at 2025-02-09 16:52:13.143905243 +0000 UTC m=+7.474137936, 105.73454ms after stability testing.go:149: cluster: 2025-02-09T16:52:13.143Z [INFO] stable state for Leader reached at 2025-02-09 16:52:13.03817071 +0000 UTC m=+7.368403396 (1 nodes), 677ns from start of poll, 196.15047ms from cluster start. Timeout at 2025-02-09 16:52:13.143905243 +0000 UTC m=+7.474137936, 105.73454ms after stability testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.144Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: updating configuration: command=AddStaging server-id=server-4786ac2c-2633-9a2e-292b-5891a4d4df85 server-addr=4786ac2c-2633-9a2e-292b-5891a4d4df85 servers="[{Suffrage:Voter ID:server-02d96105-833a-caf2-58f7-620568a4aef9 Address:02d96105-833a-caf2-58f7-620568a4aef9} {Suffrage:Voter ID:server-6b3ebfb3-99db-5bb7-8125-271801257201 Address:6b3ebfb3-99db-5bb7-8125-271801257201} {Suffrage:Voter ID:server-4786ac2c-2633-9a2e-292b-5891a4d4df85 Address:4786ac2c-2633-9a2e-292b-5891a4d4df85}]" testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.144Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: updating configuration: command=AddStaging server-id=server-4786ac2c-2633-9a2e-292b-5891a4d4df85 server-addr=4786ac2c-2633-9a2e-292b-5891a4d4df85 servers="[{Suffrage:Voter ID:server-02d96105-833a-caf2-58f7-620568a4aef9 Address:02d96105-833a-caf2-58f7-620568a4aef9} {Suffrage:Voter ID:server-6b3ebfb3-99db-5bb7-8125-271801257201 Address:6b3ebfb3-99db-5bb7-8125-271801257201} {Suffrage:Voter ID:server-4786ac2c-2633-9a2e-292b-5891a4d4df85 Address:4786ac2c-2633-9a2e-292b-5891a4d4df85}]" testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.144Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: added peer, starting replication: peer=server-4786ac2c-2633-9a2e-292b-5891a4d4df85 testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.144Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: added peer, starting replication: peer=server-4786ac2c-2633-9a2e-292b-5891a4d4df85 testLoggerAdapter verbose: server-4786ac2c-2633-9a2e-292b-5891a4d4df85: 2025-02-09T16:52:13.144Z [WARN] server-4786ac2c-2633-9a2e-292b-5891a4d4df85: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-4786ac2c-2633-9a2e-292b-5891a4d4df85: 2025-02-09T16:52:13.144Z [WARN] server-4786ac2c-2633-9a2e-292b-5891a4d4df85: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.144Z [WARN] server-02d96105-833a-caf2-58f7-620568a4aef9: appendEntries rejected, sending older logs: peer="{Voter server-4786ac2c-2633-9a2e-292b-5891a4d4df85 4786ac2c-2633-9a2e-292b-5891a4d4df85}" next=1 testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.144Z [WARN] server-02d96105-833a-caf2-58f7-620568a4aef9: appendEntries rejected, sending older logs: peer="{Voter server-4786ac2c-2633-9a2e-292b-5891a4d4df85 4786ac2c-2633-9a2e-292b-5891a4d4df85}" next=1 testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.144Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: pipelining replication: peer="{Voter server-4786ac2c-2633-9a2e-292b-5891a4d4df85 4786ac2c-2633-9a2e-292b-5891a4d4df85}" testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.144Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: pipelining replication: peer="{Voter server-4786ac2c-2633-9a2e-292b-5891a4d4df85 4786ac2c-2633-9a2e-292b-5891a4d4df85}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.144Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:13.144Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.250Z [INFO] stable state for Leader reached at 2025-02-09 16:52:13.144500799 +0000 UTC m=+7.474733486 (1 nodes), 546ns from start of poll, 302.48056ms from cluster start. Timeout at 2025-02-09 16:52:13.250350559 +0000 UTC m=+7.580583252, 105.849766ms after stability testing.go:149: cluster: 2025-02-09T16:52:13.250Z [INFO] stable state for Leader reached at 2025-02-09 16:52:13.144500799 +0000 UTC m=+7.474733486 (1 nodes), 546ns from start of poll, 302.48056ms from cluster start. Timeout at 2025-02-09 16:52:13.250350559 +0000 UTC m=+7.580583252, 105.849766ms after stability testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.250Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: aborting pipeline replication: peer="{Voter server-4786ac2c-2633-9a2e-292b-5891a4d4df85 4786ac2c-2633-9a2e-292b-5891a4d4df85}" testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.250Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: aborting pipeline replication: peer="{Voter server-4786ac2c-2633-9a2e-292b-5891a4d4df85 4786ac2c-2633-9a2e-292b-5891a4d4df85}" testLoggerAdapter verbose: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.250Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: aborting pipeline replication: peer="{Voter server-6b3ebfb3-99db-5bb7-8125-271801257201 6b3ebfb3-99db-5bb7-8125-271801257201}" testing.go:149: server-02d96105-833a-caf2-58f7-620568a4aef9: 2025-02-09T16:52:13.250Z [INFO] server-02d96105-833a-caf2-58f7-620568a4aef9: aborting pipeline replication: peer="{Voter server-6b3ebfb3-99db-5bb7-8125-271801257201 6b3ebfb3-99db-5bb7-8125-271801257201}" --- PASS: TestRaft_JoinNode_ConfigStore (0.41s) === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-78081e9e-ee36-4aa6-5a88-0e99d26443f5: 2025-02-09T16:52:13.251Z [INFO] server-78081e9e-ee36-4aa6-5a88-0e99d26443f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 Address:78081e9e-ee36-4aa6-5a88-0e99d26443f5} {Suffrage:Voter ID:server-216e7104-a45e-5dba-0d31-a2c4b44bc119 Address:216e7104-a45e-5dba-0d31-a2c4b44bc119} {Suffrage:Voter ID:server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 Address:74c9282a-aa4d-cabb-b17c-d55b9a803c09}]" testing.go:149: server-78081e9e-ee36-4aa6-5a88-0e99d26443f5: 2025-02-09T16:52:13.251Z [INFO] server-78081e9e-ee36-4aa6-5a88-0e99d26443f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 Address:78081e9e-ee36-4aa6-5a88-0e99d26443f5} {Suffrage:Voter ID:server-216e7104-a45e-5dba-0d31-a2c4b44bc119 Address:216e7104-a45e-5dba-0d31-a2c4b44bc119} {Suffrage:Voter ID:server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 Address:74c9282a-aa4d-cabb-b17c-d55b9a803c09}]" testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.251Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 Address:78081e9e-ee36-4aa6-5a88-0e99d26443f5} {Suffrage:Voter ID:server-216e7104-a45e-5dba-0d31-a2c4b44bc119 Address:216e7104-a45e-5dba-0d31-a2c4b44bc119} {Suffrage:Voter ID:server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 Address:74c9282a-aa4d-cabb-b17c-d55b9a803c09}]" testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.251Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 Address:78081e9e-ee36-4aa6-5a88-0e99d26443f5} {Suffrage:Voter ID:server-216e7104-a45e-5dba-0d31-a2c4b44bc119 Address:216e7104-a45e-5dba-0d31-a2c4b44bc119} {Suffrage:Voter ID:server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 Address:74c9282a-aa4d-cabb-b17c-d55b9a803c09}]" testLoggerAdapter verbose: server-74c9282a-aa4d-cabb-b17c-d55b9a803c09: 2025-02-09T16:52:13.251Z [INFO] server-74c9282a-aa4d-cabb-b17c-d55b9a803c09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 Address:78081e9e-ee36-4aa6-5a88-0e99d26443f5} {Suffrage:Voter ID:server-216e7104-a45e-5dba-0d31-a2c4b44bc119 Address:216e7104-a45e-5dba-0d31-a2c4b44bc119} {Suffrage:Voter ID:server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 Address:74c9282a-aa4d-cabb-b17c-d55b9a803c09}]" testing.go:149: server-74c9282a-aa4d-cabb-b17c-d55b9a803c09: 2025-02-09T16:52:13.251Z [INFO] server-74c9282a-aa4d-cabb-b17c-d55b9a803c09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 Address:78081e9e-ee36-4aa6-5a88-0e99d26443f5} {Suffrage:Voter ID:server-216e7104-a45e-5dba-0d31-a2c4b44bc119 Address:216e7104-a45e-5dba-0d31-a2c4b44bc119} {Suffrage:Voter ID:server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 Address:74c9282a-aa4d-cabb-b17c-d55b9a803c09}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.251Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:13.251Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-78081e9e-ee36-4aa6-5a88-0e99d26443f5: 2025-02-09T16:52:13.251Z [INFO] server-78081e9e-ee36-4aa6-5a88-0e99d26443f5: entering follower state: follower="Node at 78081e9e-ee36-4aa6-5a88-0e99d26443f5 [Follower]" leader= testing.go:149: server-78081e9e-ee36-4aa6-5a88-0e99d26443f5: 2025-02-09T16:52:13.251Z [INFO] server-78081e9e-ee36-4aa6-5a88-0e99d26443f5: entering follower state: follower="Node at 78081e9e-ee36-4aa6-5a88-0e99d26443f5 [Follower]" leader= testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.251Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: entering follower state: follower="Node at 216e7104-a45e-5dba-0d31-a2c4b44bc119 [Follower]" leader= testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.251Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: entering follower state: follower="Node at 216e7104-a45e-5dba-0d31-a2c4b44bc119 [Follower]" leader= testLoggerAdapter verbose: server-74c9282a-aa4d-cabb-b17c-d55b9a803c09: 2025-02-09T16:52:13.251Z [INFO] server-74c9282a-aa4d-cabb-b17c-d55b9a803c09: entering follower state: follower="Node at 74c9282a-aa4d-cabb-b17c-d55b9a803c09 [Follower]" leader= testing.go:149: server-74c9282a-aa4d-cabb-b17c-d55b9a803c09: 2025-02-09T16:52:13.251Z [INFO] server-74c9282a-aa4d-cabb-b17c-d55b9a803c09: entering follower state: follower="Node at 74c9282a-aa4d-cabb-b17c-d55b9a803c09 [Follower]" leader= testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.321Z [WARN] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.321Z [WARN] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.321Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: entering candidate state: node="Node at 216e7104-a45e-5dba-0d31-a2c4b44bc119 [Candidate]" term=2 testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.321Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: entering candidate state: node="Node at 216e7104-a45e-5dba-0d31-a2c4b44bc119 [Candidate]" term=2 testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: election won: tally=2 testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: election won: tally=2 testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: entering leader state: leader="Node at 216e7104-a45e-5dba-0d31-a2c4b44bc119 [Leader]" testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: entering leader state: leader="Node at 216e7104-a45e-5dba-0d31-a2c4b44bc119 [Leader]" testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: added peer, starting replication: peer=server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: added peer, starting replication: peer=server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: added peer, starting replication: peer=server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: added peer, starting replication: peer=server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: pipelining replication: peer="{Voter server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 74c9282a-aa4d-cabb-b17c-d55b9a803c09}" testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: pipelining replication: peer="{Voter server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 74c9282a-aa4d-cabb-b17c-d55b9a803c09}" testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: pipelining replication: peer="{Voter server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 78081e9e-ee36-4aa6-5a88-0e99d26443f5}" testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.322Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: pipelining replication: peer="{Voter server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 78081e9e-ee36-4aa6-5a88-0e99d26443f5}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.428Z [INFO] stable state for Leader reached at 2025-02-09 16:52:13.322467147 +0000 UTC m=+7.652699837 (1 nodes), 70.809037ms from start of poll, 71.249206ms from cluster start. Timeout at 2025-02-09 16:52:13.428412051 +0000 UTC m=+7.758644742, 105.944905ms after stability testing.go:149: cluster: 2025-02-09T16:52:13.428Z [INFO] stable state for Leader reached at 2025-02-09 16:52:13.322467147 +0000 UTC m=+7.652699837 (1 nodes), 70.809037ms from start of poll, 71.249206ms from cluster start. Timeout at 2025-02-09 16:52:13.428412051 +0000 UTC m=+7.758644742, 105.944905ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.434Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:13.434Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.539Z [INFO] stable state for Follower reached at 2025-02-09 16:52:13.434166821 +0000 UTC m=+7.764399507 (2 nodes), 1.078µs from start of poll, 182.948876ms from cluster start. Timeout at 2025-02-09 16:52:13.539167788 +0000 UTC m=+7.869400484, 105.000977ms after stability testing.go:149: cluster: 2025-02-09T16:52:13.539Z [INFO] stable state for Follower reached at 2025-02-09 16:52:13.434166821 +0000 UTC m=+7.764399507 (2 nodes), 1.078µs from start of poll, 182.948876ms from cluster start. Timeout at 2025-02-09 16:52:13.539167788 +0000 UTC m=+7.869400484, 105.000977ms after stability testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.539Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: updating configuration: command=RemoveServer server-id=server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 server-addr= servers="[{Suffrage:Voter ID:server-216e7104-a45e-5dba-0d31-a2c4b44bc119 Address:216e7104-a45e-5dba-0d31-a2c4b44bc119} {Suffrage:Voter ID:server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 Address:74c9282a-aa4d-cabb-b17c-d55b9a803c09}]" testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.539Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: updating configuration: command=RemoveServer server-id=server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 server-addr= servers="[{Suffrage:Voter ID:server-216e7104-a45e-5dba-0d31-a2c4b44bc119 Address:216e7104-a45e-5dba-0d31-a2c4b44bc119} {Suffrage:Voter ID:server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 Address:74c9282a-aa4d-cabb-b17c-d55b9a803c09}]" testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.539Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: removed peer, stopping replication: peer=server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 last-index=3 testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.539Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: removed peer, stopping replication: peer=server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 last-index=3 testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.539Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: aborting pipeline replication: peer="{Voter server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 78081e9e-ee36-4aa6-5a88-0e99d26443f5}" testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.539Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: aborting pipeline replication: peer="{Voter server-78081e9e-ee36-4aa6-5a88-0e99d26443f5 78081e9e-ee36-4aa6-5a88-0e99d26443f5}" testLoggerAdapter verbose: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.644Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: aborting pipeline replication: peer="{Voter server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 74c9282a-aa4d-cabb-b17c-d55b9a803c09}" testing.go:149: server-216e7104-a45e-5dba-0d31-a2c4b44bc119: 2025-02-09T16:52:13.644Z [INFO] server-216e7104-a45e-5dba-0d31-a2c4b44bc119: aborting pipeline replication: peer="{Voter server-74c9282a-aa4d-cabb-b17c-d55b9a803c09 74c9282a-aa4d-cabb-b17c-d55b9a803c09}" --- PASS: TestRaft_RemoveFollower (0.39s) === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.645Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25624f96-c9b5-cd5e-601a-eb7e239f288a Address:25624f96-c9b5-cd5e-601a-eb7e239f288a} {Suffrage:Voter ID:server-e302a44c-9970-c5d6-a626-74e3f3d554e2 Address:e302a44c-9970-c5d6-a626-74e3f3d554e2} {Suffrage:Voter ID:server-36cfb538-214d-4985-fd24-19cfc744b5d8 Address:36cfb538-214d-4985-fd24-19cfc744b5d8}]" testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.645Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25624f96-c9b5-cd5e-601a-eb7e239f288a Address:25624f96-c9b5-cd5e-601a-eb7e239f288a} {Suffrage:Voter ID:server-e302a44c-9970-c5d6-a626-74e3f3d554e2 Address:e302a44c-9970-c5d6-a626-74e3f3d554e2} {Suffrage:Voter ID:server-36cfb538-214d-4985-fd24-19cfc744b5d8 Address:36cfb538-214d-4985-fd24-19cfc744b5d8}]" testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:13.645Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25624f96-c9b5-cd5e-601a-eb7e239f288a Address:25624f96-c9b5-cd5e-601a-eb7e239f288a} {Suffrage:Voter ID:server-e302a44c-9970-c5d6-a626-74e3f3d554e2 Address:e302a44c-9970-c5d6-a626-74e3f3d554e2} {Suffrage:Voter ID:server-36cfb538-214d-4985-fd24-19cfc744b5d8 Address:36cfb538-214d-4985-fd24-19cfc744b5d8}]" testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:13.645Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25624f96-c9b5-cd5e-601a-eb7e239f288a Address:25624f96-c9b5-cd5e-601a-eb7e239f288a} {Suffrage:Voter ID:server-e302a44c-9970-c5d6-a626-74e3f3d554e2 Address:e302a44c-9970-c5d6-a626-74e3f3d554e2} {Suffrage:Voter ID:server-36cfb538-214d-4985-fd24-19cfc744b5d8 Address:36cfb538-214d-4985-fd24-19cfc744b5d8}]" testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.645Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25624f96-c9b5-cd5e-601a-eb7e239f288a Address:25624f96-c9b5-cd5e-601a-eb7e239f288a} {Suffrage:Voter ID:server-e302a44c-9970-c5d6-a626-74e3f3d554e2 Address:e302a44c-9970-c5d6-a626-74e3f3d554e2} {Suffrage:Voter ID:server-36cfb538-214d-4985-fd24-19cfc744b5d8 Address:36cfb538-214d-4985-fd24-19cfc744b5d8}]" testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.645Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25624f96-c9b5-cd5e-601a-eb7e239f288a Address:25624f96-c9b5-cd5e-601a-eb7e239f288a} {Suffrage:Voter ID:server-e302a44c-9970-c5d6-a626-74e3f3d554e2 Address:e302a44c-9970-c5d6-a626-74e3f3d554e2} {Suffrage:Voter ID:server-36cfb538-214d-4985-fd24-19cfc744b5d8 Address:36cfb538-214d-4985-fd24-19cfc744b5d8}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.645Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:13.645Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.645Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: entering follower state: follower="Node at 25624f96-c9b5-cd5e-601a-eb7e239f288a [Follower]" leader= testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.645Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: entering follower state: follower="Node at 25624f96-c9b5-cd5e-601a-eb7e239f288a [Follower]" leader= testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:13.645Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: entering follower state: follower="Node at e302a44c-9970-c5d6-a626-74e3f3d554e2 [Follower]" leader= testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:13.645Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: entering follower state: follower="Node at e302a44c-9970-c5d6-a626-74e3f3d554e2 [Follower]" leader= testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.645Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: entering follower state: follower="Node at 36cfb538-214d-4985-fd24-19cfc744b5d8 [Follower]" leader= testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.645Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: entering follower state: follower="Node at 36cfb538-214d-4985-fd24-19cfc744b5d8 [Follower]" leader= testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [WARN] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [WARN] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: entering candidate state: node="Node at 25624f96-c9b5-cd5e-601a-eb7e239f288a [Candidate]" term=2 testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: entering candidate state: node="Node at 25624f96-c9b5-cd5e-601a-eb7e239f288a [Candidate]" term=2 testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.702Z [WARN] server-36cfb538-214d-4985-fd24-19cfc744b5d8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.702Z [WARN] server-36cfb538-214d-4985-fd24-19cfc744b5d8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.702Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: entering candidate state: node="Node at 36cfb538-214d-4985-fd24-19cfc744b5d8 [Candidate]" term=2 testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.702Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: entering candidate state: node="Node at 36cfb538-214d-4985-fd24-19cfc744b5d8 [Candidate]" term=2 testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.702Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: duplicate requestVote for same term: term=2 testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.702Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: election won: tally=2 testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: election won: tally=2 testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: entering leader state: leader="Node at 25624f96-c9b5-cd5e-601a-eb7e239f288a [Leader]" testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: entering leader state: leader="Node at 25624f96-c9b5-cd5e-601a-eb7e239f288a [Leader]" testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: added peer, starting replication: peer=server-e302a44c-9970-c5d6-a626-74e3f3d554e2 testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: added peer, starting replication: peer=server-e302a44c-9970-c5d6-a626-74e3f3d554e2 testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: added peer, starting replication: peer=server-36cfb538-214d-4985-fd24-19cfc744b5d8 testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.702Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: added peer, starting replication: peer=server-36cfb538-214d-4985-fd24-19cfc744b5d8 testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.703Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: entering follower state: follower="Node at 36cfb538-214d-4985-fd24-19cfc744b5d8 [Follower]" leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.703Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: entering follower state: follower="Node at 36cfb538-214d-4985-fd24-19cfc744b5d8 [Follower]" leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.703Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: pipelining replication: peer="{Voter server-36cfb538-214d-4985-fd24-19cfc744b5d8 36cfb538-214d-4985-fd24-19cfc744b5d8}" testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.703Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: pipelining replication: peer="{Voter server-36cfb538-214d-4985-fd24-19cfc744b5d8 36cfb538-214d-4985-fd24-19cfc744b5d8}" testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:13.703Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: duplicate requestVote for same term: term=2 testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:13.703Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.703Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: pipelining replication: peer="{Voter server-e302a44c-9970-c5d6-a626-74e3f3d554e2 e302a44c-9970-c5d6-a626-74e3f3d554e2}" testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.703Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: pipelining replication: peer="{Voter server-e302a44c-9970-c5d6-a626-74e3f3d554e2 e302a44c-9970-c5d6-a626-74e3f3d554e2}" testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.703Z [WARN] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: rejecting vote request since we have a leader: from=36cfb538-214d-4985-fd24-19cfc744b5d8 leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.703Z [WARN] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: rejecting vote request since we have a leader: from=36cfb538-214d-4985-fd24-19cfc744b5d8 leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.808Z [INFO] stable state for Leader reached at 2025-02-09 16:52:13.703262206 +0000 UTC m=+8.033494894 (1 nodes), 57.459445ms from start of poll, 57.875472ms from cluster start. Timeout at 2025-02-09 16:52:13.808901504 +0000 UTC m=+8.139134192, 105.639298ms after stability testing.go:149: cluster: 2025-02-09T16:52:13.808Z [INFO] stable state for Leader reached at 2025-02-09 16:52:13.703262206 +0000 UTC m=+8.033494894 (1 nodes), 57.459445ms from start of poll, 57.875472ms from cluster start. Timeout at 2025-02-09 16:52:13.808901504 +0000 UTC m=+8.139134192, 105.639298ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.814Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:13.814Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:13.920Z [INFO] stable state for Follower reached at 2025-02-09 16:52:13.814621126 +0000 UTC m=+8.144853812 (2 nodes), 994ns from start of poll, 169.23439ms from cluster start. Timeout at 2025-02-09 16:52:13.920272331 +0000 UTC m=+8.250505019, 105.651207ms after stability testing.go:149: cluster: 2025-02-09T16:52:13.920Z [INFO] stable state for Follower reached at 2025-02-09 16:52:13.814621126 +0000 UTC m=+8.144853812 (2 nodes), 994ns from start of poll, 169.23439ms from cluster start. Timeout at 2025-02-09 16:52:13.920272331 +0000 UTC m=+8.250505019, 105.651207ms after stability testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.920Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: updating configuration: command=RemoveServer server-id=server-25624f96-c9b5-cd5e-601a-eb7e239f288a server-addr= servers="[{Suffrage:Voter ID:server-e302a44c-9970-c5d6-a626-74e3f3d554e2 Address:e302a44c-9970-c5d6-a626-74e3f3d554e2} {Suffrage:Voter ID:server-36cfb538-214d-4985-fd24-19cfc744b5d8 Address:36cfb538-214d-4985-fd24-19cfc744b5d8}]" testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.920Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: updating configuration: command=RemoveServer server-id=server-25624f96-c9b5-cd5e-601a-eb7e239f288a server-addr= servers="[{Suffrage:Voter ID:server-e302a44c-9970-c5d6-a626-74e3f3d554e2 Address:e302a44c-9970-c5d6-a626-74e3f3d554e2} {Suffrage:Voter ID:server-36cfb538-214d-4985-fd24-19cfc744b5d8 Address:36cfb538-214d-4985-fd24-19cfc744b5d8}]" testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.920Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: removed ourself, shutting down testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.920Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: removed ourself, shutting down testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.920Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: aborting pipeline replication: peer="{Voter server-36cfb538-214d-4985-fd24-19cfc744b5d8 36cfb538-214d-4985-fd24-19cfc744b5d8}" testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.920Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: aborting pipeline replication: peer="{Voter server-36cfb538-214d-4985-fd24-19cfc744b5d8 36cfb538-214d-4985-fd24-19cfc744b5d8}" testLoggerAdapter verbose: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.920Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: aborting pipeline replication: peer="{Voter server-e302a44c-9970-c5d6-a626-74e3f3d554e2 e302a44c-9970-c5d6-a626-74e3f3d554e2}" testing.go:149: server-25624f96-c9b5-cd5e-601a-eb7e239f288a: 2025-02-09T16:52:13.920Z [INFO] server-25624f96-c9b5-cd5e-601a-eb7e239f288a: aborting pipeline replication: peer="{Voter server-e302a44c-9970-c5d6-a626-74e3f3d554e2 e302a44c-9970-c5d6-a626-74e3f3d554e2}" testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:13.975Z [WARN] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: heartbeat timeout reached, starting election: last-leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:13.975Z [WARN] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: heartbeat timeout reached, starting election: last-leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:13.975Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: entering candidate state: node="Node at e302a44c-9970-c5d6-a626-74e3f3d554e2 [Candidate]" term=3 testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:13.975Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: entering candidate state: node="Node at e302a44c-9970-c5d6-a626-74e3f3d554e2 [Candidate]" term=3 testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.975Z [WARN] server-36cfb538-214d-4985-fd24-19cfc744b5d8: rejecting vote request since we have a leader: from=e302a44c-9970-c5d6-a626-74e3f3d554e2 leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:13.975Z [WARN] server-36cfb538-214d-4985-fd24-19cfc744b5d8: rejecting vote request since we have a leader: from=e302a44c-9970-c5d6-a626-74e3f3d554e2 leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.033Z [WARN] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: Election timeout reached, restarting election testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.033Z [WARN] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: Election timeout reached, restarting election testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.033Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: entering candidate state: node="Node at e302a44c-9970-c5d6-a626-74e3f3d554e2 [Candidate]" term=4 testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.033Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: entering candidate state: node="Node at e302a44c-9970-c5d6-a626-74e3f3d554e2 [Candidate]" term=4 testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:14.033Z [WARN] server-36cfb538-214d-4985-fd24-19cfc744b5d8: rejecting vote request since we have a leader: from=e302a44c-9970-c5d6-a626-74e3f3d554e2 leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:14.033Z [WARN] server-36cfb538-214d-4985-fd24-19cfc744b5d8: rejecting vote request since we have a leader: from=e302a44c-9970-c5d6-a626-74e3f3d554e2 leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:14.034Z [WARN] server-36cfb538-214d-4985-fd24-19cfc744b5d8: heartbeat timeout reached, starting election: last-leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:14.034Z [WARN] server-36cfb538-214d-4985-fd24-19cfc744b5d8: heartbeat timeout reached, starting election: last-leader=25624f96-c9b5-cd5e-601a-eb7e239f288a testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:14.034Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: entering candidate state: node="Node at 36cfb538-214d-4985-fd24-19cfc744b5d8 [Candidate]" term=3 testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:14.034Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: entering candidate state: node="Node at 36cfb538-214d-4985-fd24-19cfc744b5d8 [Candidate]" term=3 testLoggerAdapter verbose: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:14.034Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: entering follower state: follower="Node at 36cfb538-214d-4985-fd24-19cfc744b5d8 [Follower]" leader= testing.go:149: server-36cfb538-214d-4985-fd24-19cfc744b5d8: 2025-02-09T16:52:14.034Z [INFO] server-36cfb538-214d-4985-fd24-19cfc744b5d8: entering follower state: follower="Node at 36cfb538-214d-4985-fd24-19cfc744b5d8 [Follower]" leader= testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [WARN] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: Election timeout reached, restarting election testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [WARN] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: Election timeout reached, restarting election testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: entering candidate state: node="Node at e302a44c-9970-c5d6-a626-74e3f3d554e2 [Candidate]" term=5 testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: entering candidate state: node="Node at e302a44c-9970-c5d6-a626-74e3f3d554e2 [Candidate]" term=5 testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: election won: tally=2 testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: election won: tally=2 testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: entering leader state: leader="Node at e302a44c-9970-c5d6-a626-74e3f3d554e2 [Leader]" testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: entering leader state: leader="Node at e302a44c-9970-c5d6-a626-74e3f3d554e2 [Leader]" testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: added peer, starting replication: peer=server-36cfb538-214d-4985-fd24-19cfc744b5d8 testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: added peer, starting replication: peer=server-36cfb538-214d-4985-fd24-19cfc744b5d8 testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: pipelining replication: peer="{Voter server-36cfb538-214d-4985-fd24-19cfc744b5d8 36cfb538-214d-4985-fd24-19cfc744b5d8}" testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.087Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: pipelining replication: peer="{Voter server-36cfb538-214d-4985-fd24-19cfc744b5d8 36cfb538-214d-4985-fd24-19cfc744b5d8}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:14.131Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:14.131Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:14.237Z [INFO] stable state for Leader reached at 2025-02-09 16:52:14.131539506 +0000 UTC m=+8.461772209 (1 nodes), 34.405µs from start of poll, 486.152787ms from cluster start. Timeout at 2025-02-09 16:52:14.237360643 +0000 UTC m=+8.567593331, 105.821122ms after stability testing.go:149: cluster: 2025-02-09T16:52:14.237Z [INFO] stable state for Leader reached at 2025-02-09 16:52:14.131539506 +0000 UTC m=+8.461772209 (1 nodes), 34.405µs from start of poll, 486.152787ms from cluster start. Timeout at 2025-02-09 16:52:14.237360643 +0000 UTC m=+8.567593331, 105.821122ms after stability testLoggerAdapter verbose: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.237Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: aborting pipeline replication: peer="{Voter server-36cfb538-214d-4985-fd24-19cfc744b5d8 36cfb538-214d-4985-fd24-19cfc744b5d8}" testing.go:149: server-e302a44c-9970-c5d6-a626-74e3f3d554e2: 2025-02-09T16:52:14.237Z [INFO] server-e302a44c-9970-c5d6-a626-74e3f3d554e2: aborting pipeline replication: peer="{Voter server-36cfb538-214d-4985-fd24-19cfc744b5d8 36cfb538-214d-4985-fd24-19cfc744b5d8}" --- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.238Z [INFO] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-921b87d6-5ba1-cc98-59c6-b2d8976d315a Address:921b87d6-5ba1-cc98-59c6-b2d8976d315a} {Suffrage:Voter ID:server-09107ffa-a86a-efad-333b-a373ce69d851 Address:09107ffa-a86a-efad-333b-a373ce69d851} {Suffrage:Voter ID:server-3b3cecc4-6229-20db-0d25-2574566cc514 Address:3b3cecc4-6229-20db-0d25-2574566cc514}]" testing.go:149: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.238Z [INFO] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-921b87d6-5ba1-cc98-59c6-b2d8976d315a Address:921b87d6-5ba1-cc98-59c6-b2d8976d315a} {Suffrage:Voter ID:server-09107ffa-a86a-efad-333b-a373ce69d851 Address:09107ffa-a86a-efad-333b-a373ce69d851} {Suffrage:Voter ID:server-3b3cecc4-6229-20db-0d25-2574566cc514 Address:3b3cecc4-6229-20db-0d25-2574566cc514}]" testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.238Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-921b87d6-5ba1-cc98-59c6-b2d8976d315a Address:921b87d6-5ba1-cc98-59c6-b2d8976d315a} {Suffrage:Voter ID:server-09107ffa-a86a-efad-333b-a373ce69d851 Address:09107ffa-a86a-efad-333b-a373ce69d851} {Suffrage:Voter ID:server-3b3cecc4-6229-20db-0d25-2574566cc514 Address:3b3cecc4-6229-20db-0d25-2574566cc514}]" testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.238Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-921b87d6-5ba1-cc98-59c6-b2d8976d315a Address:921b87d6-5ba1-cc98-59c6-b2d8976d315a} {Suffrage:Voter ID:server-09107ffa-a86a-efad-333b-a373ce69d851 Address:09107ffa-a86a-efad-333b-a373ce69d851} {Suffrage:Voter ID:server-3b3cecc4-6229-20db-0d25-2574566cc514 Address:3b3cecc4-6229-20db-0d25-2574566cc514}]" testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.238Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-921b87d6-5ba1-cc98-59c6-b2d8976d315a Address:921b87d6-5ba1-cc98-59c6-b2d8976d315a} {Suffrage:Voter ID:server-09107ffa-a86a-efad-333b-a373ce69d851 Address:09107ffa-a86a-efad-333b-a373ce69d851} {Suffrage:Voter ID:server-3b3cecc4-6229-20db-0d25-2574566cc514 Address:3b3cecc4-6229-20db-0d25-2574566cc514}]" testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.238Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-921b87d6-5ba1-cc98-59c6-b2d8976d315a Address:921b87d6-5ba1-cc98-59c6-b2d8976d315a} {Suffrage:Voter ID:server-09107ffa-a86a-efad-333b-a373ce69d851 Address:09107ffa-a86a-efad-333b-a373ce69d851} {Suffrage:Voter ID:server-3b3cecc4-6229-20db-0d25-2574566cc514 Address:3b3cecc4-6229-20db-0d25-2574566cc514}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:14.238Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:14.238Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.238Z [INFO] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: entering follower state: follower="Node at 921b87d6-5ba1-cc98-59c6-b2d8976d315a [Follower]" leader= testing.go:149: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.238Z [INFO] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: entering follower state: follower="Node at 921b87d6-5ba1-cc98-59c6-b2d8976d315a [Follower]" leader= testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.238Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: entering follower state: follower="Node at 09107ffa-a86a-efad-333b-a373ce69d851 [Follower]" leader= testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.238Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: entering follower state: follower="Node at 09107ffa-a86a-efad-333b-a373ce69d851 [Follower]" leader= testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.238Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: entering follower state: follower="Node at 3b3cecc4-6229-20db-0d25-2574566cc514 [Follower]" leader= testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.238Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: entering follower state: follower="Node at 3b3cecc4-6229-20db-0d25-2574566cc514 [Follower]" leader= testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [WARN] server-3b3cecc4-6229-20db-0d25-2574566cc514: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [WARN] server-3b3cecc4-6229-20db-0d25-2574566cc514: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: entering candidate state: node="Node at 3b3cecc4-6229-20db-0d25-2574566cc514 [Candidate]" term=2 testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: entering candidate state: node="Node at 3b3cecc4-6229-20db-0d25-2574566cc514 [Candidate]" term=2 testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: election won: tally=2 testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: election won: tally=2 testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: entering leader state: leader="Node at 3b3cecc4-6229-20db-0d25-2574566cc514 [Leader]" testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: entering leader state: leader="Node at 3b3cecc4-6229-20db-0d25-2574566cc514 [Leader]" testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: added peer, starting replication: peer=server-921b87d6-5ba1-cc98-59c6-b2d8976d315a testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: added peer, starting replication: peer=server-921b87d6-5ba1-cc98-59c6-b2d8976d315a testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: added peer, starting replication: peer=server-09107ffa-a86a-efad-333b-a373ce69d851 testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: added peer, starting replication: peer=server-09107ffa-a86a-efad-333b-a373ce69d851 testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: pipelining replication: peer="{Voter server-09107ffa-a86a-efad-333b-a373ce69d851 09107ffa-a86a-efad-333b-a373ce69d851}" testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: pipelining replication: peer="{Voter server-09107ffa-a86a-efad-333b-a373ce69d851 09107ffa-a86a-efad-333b-a373ce69d851}" testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: pipelining replication: peer="{Voter server-921b87d6-5ba1-cc98-59c6-b2d8976d315a 921b87d6-5ba1-cc98-59c6-b2d8976d315a}" testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.298Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: pipelining replication: peer="{Voter server-921b87d6-5ba1-cc98-59c6-b2d8976d315a 921b87d6-5ba1-cc98-59c6-b2d8976d315a}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:14.404Z [INFO] stable state for Follower reached at 2025-02-09 16:52:14.299047851 +0000 UTC m=+8.629280539 (2 nodes), 60.539256ms from start of poll, 60.933911ms from cluster start. Timeout at 2025-02-09 16:52:14.404350543 +0000 UTC m=+8.734583229, 105.30269ms after stability testing.go:149: cluster: 2025-02-09T16:52:14.404Z [INFO] stable state for Follower reached at 2025-02-09 16:52:14.299047851 +0000 UTC m=+8.629280539 (2 nodes), 60.539256ms from start of poll, 60.933911ms from cluster start. Timeout at 2025-02-09 16:52:14.404350543 +0000 UTC m=+8.734583229, 105.30269ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:14.404Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:14.404Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:14.510Z [INFO] stable state for Leader reached at 2025-02-09 16:52:14.404486871 +0000 UTC m=+8.734719557 (1 nodes), 1.234µs from start of poll, 166.372929ms from cluster start. Timeout at 2025-02-09 16:52:14.510232476 +0000 UTC m=+8.840465162, 105.745605ms after stability testing.go:149: cluster: 2025-02-09T16:52:14.510Z [INFO] stable state for Leader reached at 2025-02-09 16:52:14.404486871 +0000 UTC m=+8.734719557 (1 nodes), 1.234µs from start of poll, 166.372929ms from cluster start. Timeout at 2025-02-09 16:52:14.510232476 +0000 UTC m=+8.840465162, 105.745605ms after stability testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.514Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: updating configuration: command=RemoveServer server-id=server-3b3cecc4-6229-20db-0d25-2574566cc514 server-addr= servers="[{Suffrage:Voter ID:server-921b87d6-5ba1-cc98-59c6-b2d8976d315a Address:921b87d6-5ba1-cc98-59c6-b2d8976d315a} {Suffrage:Voter ID:server-09107ffa-a86a-efad-333b-a373ce69d851 Address:09107ffa-a86a-efad-333b-a373ce69d851}]" testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.514Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: updating configuration: command=RemoveServer server-id=server-3b3cecc4-6229-20db-0d25-2574566cc514 server-addr= servers="[{Suffrage:Voter ID:server-921b87d6-5ba1-cc98-59c6-b2d8976d315a Address:921b87d6-5ba1-cc98-59c6-b2d8976d315a} {Suffrage:Voter ID:server-09107ffa-a86a-efad-333b-a373ce69d851 Address:09107ffa-a86a-efad-333b-a373ce69d851}]" testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.515Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: removed ourself, transitioning to follower testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.515Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: removed ourself, transitioning to follower testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.516Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: entering follower state: follower="Node at 3b3cecc4-6229-20db-0d25-2574566cc514 [Follower]" leader= testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.516Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: entering follower state: follower="Node at 3b3cecc4-6229-20db-0d25-2574566cc514 [Follower]" leader= testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.516Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: aborting pipeline replication: peer="{Voter server-09107ffa-a86a-efad-333b-a373ce69d851 09107ffa-a86a-efad-333b-a373ce69d851}" testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.516Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: aborting pipeline replication: peer="{Voter server-09107ffa-a86a-efad-333b-a373ce69d851 09107ffa-a86a-efad-333b-a373ce69d851}" testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.516Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: aborting pipeline replication: peer="{Voter server-921b87d6-5ba1-cc98-59c6-b2d8976d315a 921b87d6-5ba1-cc98-59c6-b2d8976d315a}" testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.516Z [INFO] server-3b3cecc4-6229-20db-0d25-2574566cc514: aborting pipeline replication: peer="{Voter server-921b87d6-5ba1-cc98-59c6-b2d8976d315a 921b87d6-5ba1-cc98-59c6-b2d8976d315a}" testLoggerAdapter verbose: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.586Z [WARN] server-3b3cecc4-6229-20db-0d25-2574566cc514: not part of stable configuration, aborting election testing.go:149: server-3b3cecc4-6229-20db-0d25-2574566cc514: 2025-02-09T16:52:14.586Z [WARN] server-3b3cecc4-6229-20db-0d25-2574566cc514: not part of stable configuration, aborting election testLoggerAdapter verbose: cluster: 2025-02-09T16:52:14.622Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:14.622Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.627Z [WARN] server-09107ffa-a86a-efad-333b-a373ce69d851: heartbeat timeout reached, starting election: last-leader=3b3cecc4-6229-20db-0d25-2574566cc514 testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.627Z [WARN] server-09107ffa-a86a-efad-333b-a373ce69d851: heartbeat timeout reached, starting election: last-leader=3b3cecc4-6229-20db-0d25-2574566cc514 testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.627Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: entering candidate state: node="Node at 09107ffa-a86a-efad-333b-a373ce69d851 [Candidate]" term=3 testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.627Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: entering candidate state: node="Node at 09107ffa-a86a-efad-333b-a373ce69d851 [Candidate]" term=3 testLoggerAdapter verbose: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.627Z [WARN] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: rejecting vote request since we have a leader: from=09107ffa-a86a-efad-333b-a373ce69d851 leader=3b3cecc4-6229-20db-0d25-2574566cc514 testing.go:149: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.627Z [WARN] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: rejecting vote request since we have a leader: from=09107ffa-a86a-efad-333b-a373ce69d851 leader=3b3cecc4-6229-20db-0d25-2574566cc514 testLoggerAdapter verbose: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.645Z [WARN] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: heartbeat timeout reached, starting election: last-leader=3b3cecc4-6229-20db-0d25-2574566cc514 testing.go:149: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.645Z [WARN] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: heartbeat timeout reached, starting election: last-leader=3b3cecc4-6229-20db-0d25-2574566cc514 testLoggerAdapter verbose: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.645Z [INFO] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: entering candidate state: node="Node at 921b87d6-5ba1-cc98-59c6-b2d8976d315a [Candidate]" term=3 testing.go:149: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.645Z [INFO] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: entering candidate state: node="Node at 921b87d6-5ba1-cc98-59c6-b2d8976d315a [Candidate]" term=3 testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.645Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: duplicate requestVote for same term: term=3 testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.645Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.714Z [WARN] server-09107ffa-a86a-efad-333b-a373ce69d851: Election timeout reached, restarting election testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.714Z [WARN] server-09107ffa-a86a-efad-333b-a373ce69d851: Election timeout reached, restarting election testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.714Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: entering candidate state: node="Node at 09107ffa-a86a-efad-333b-a373ce69d851 [Candidate]" term=4 testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.714Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: entering candidate state: node="Node at 09107ffa-a86a-efad-333b-a373ce69d851 [Candidate]" term=4 testLoggerAdapter verbose: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.714Z [INFO] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: entering follower state: follower="Node at 921b87d6-5ba1-cc98-59c6-b2d8976d315a [Follower]" leader= testing.go:149: server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: 2025-02-09T16:52:14.714Z [INFO] server-921b87d6-5ba1-cc98-59c6-b2d8976d315a: entering follower state: follower="Node at 921b87d6-5ba1-cc98-59c6-b2d8976d315a [Follower]" leader= testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.714Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: election won: tally=2 testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.714Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: election won: tally=2 testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.714Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: entering leader state: leader="Node at 09107ffa-a86a-efad-333b-a373ce69d851 [Leader]" testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.714Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: entering leader state: leader="Node at 09107ffa-a86a-efad-333b-a373ce69d851 [Leader]" testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.714Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: added peer, starting replication: peer=server-921b87d6-5ba1-cc98-59c6-b2d8976d315a testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.714Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: added peer, starting replication: peer=server-921b87d6-5ba1-cc98-59c6-b2d8976d315a testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.715Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: pipelining replication: peer="{Voter server-921b87d6-5ba1-cc98-59c6-b2d8976d315a 921b87d6-5ba1-cc98-59c6-b2d8976d315a}" testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.715Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: pipelining replication: peer="{Voter server-921b87d6-5ba1-cc98-59c6-b2d8976d315a 921b87d6-5ba1-cc98-59c6-b2d8976d315a}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:14.821Z [INFO] stable state for Leader reached at 2025-02-09 16:52:14.715126831 +0000 UTC m=+9.045359519 (1 nodes), 92.713542ms from start of poll, 477.012891ms from cluster start. Timeout at 2025-02-09 16:52:14.821153953 +0000 UTC m=+9.151386639, 106.02712ms after stability testing.go:149: cluster: 2025-02-09T16:52:14.821Z [INFO] stable state for Leader reached at 2025-02-09 16:52:14.715126831 +0000 UTC m=+9.045359519 (1 nodes), 92.713542ms from start of poll, 477.012891ms from cluster start. Timeout at 2025-02-09 16:52:14.821153953 +0000 UTC m=+9.151386639, 106.02712ms after stability testLoggerAdapter verbose: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.927Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: aborting pipeline replication: peer="{Voter server-921b87d6-5ba1-cc98-59c6-b2d8976d315a 921b87d6-5ba1-cc98-59c6-b2d8976d315a}" testing.go:149: server-09107ffa-a86a-efad-333b-a373ce69d851: 2025-02-09T16:52:14.927Z [INFO] server-09107ffa-a86a-efad-333b-a373ce69d851: aborting pipeline replication: peer="{Voter server-921b87d6-5ba1-cc98-59c6-b2d8976d315a 921b87d6-5ba1-cc98-59c6-b2d8976d315a}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.69s) === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-c3e38f57-e3c1-594f-6781-22113fe78d22: 2025-02-09T16:52:14.927Z [INFO] server-c3e38f57-e3c1-594f-6781-22113fe78d22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3e38f57-e3c1-594f-6781-22113fe78d22 Address:c3e38f57-e3c1-594f-6781-22113fe78d22} {Suffrage:Voter ID:server-09770b7f-cd60-a691-7d9c-d702b0609b28 Address:09770b7f-cd60-a691-7d9c-d702b0609b28} {Suffrage:Voter ID:server-19e4abba-dfde-6448-4f48-bf7641c8f371 Address:19e4abba-dfde-6448-4f48-bf7641c8f371} {Suffrage:Voter ID:server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 Address:293b25f1-82fb-cd6d-7127-8cdaa60a4e22}]" testing.go:149: server-c3e38f57-e3c1-594f-6781-22113fe78d22: 2025-02-09T16:52:14.927Z [INFO] server-c3e38f57-e3c1-594f-6781-22113fe78d22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3e38f57-e3c1-594f-6781-22113fe78d22 Address:c3e38f57-e3c1-594f-6781-22113fe78d22} {Suffrage:Voter ID:server-09770b7f-cd60-a691-7d9c-d702b0609b28 Address:09770b7f-cd60-a691-7d9c-d702b0609b28} {Suffrage:Voter ID:server-19e4abba-dfde-6448-4f48-bf7641c8f371 Address:19e4abba-dfde-6448-4f48-bf7641c8f371} {Suffrage:Voter ID:server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 Address:293b25f1-82fb-cd6d-7127-8cdaa60a4e22}]" testLoggerAdapter verbose: server-09770b7f-cd60-a691-7d9c-d702b0609b28: 2025-02-09T16:52:14.928Z [INFO] server-09770b7f-cd60-a691-7d9c-d702b0609b28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3e38f57-e3c1-594f-6781-22113fe78d22 Address:c3e38f57-e3c1-594f-6781-22113fe78d22} {Suffrage:Voter ID:server-09770b7f-cd60-a691-7d9c-d702b0609b28 Address:09770b7f-cd60-a691-7d9c-d702b0609b28} {Suffrage:Voter ID:server-19e4abba-dfde-6448-4f48-bf7641c8f371 Address:19e4abba-dfde-6448-4f48-bf7641c8f371} {Suffrage:Voter ID:server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 Address:293b25f1-82fb-cd6d-7127-8cdaa60a4e22}]" testing.go:149: server-09770b7f-cd60-a691-7d9c-d702b0609b28: 2025-02-09T16:52:14.928Z [INFO] server-09770b7f-cd60-a691-7d9c-d702b0609b28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3e38f57-e3c1-594f-6781-22113fe78d22 Address:c3e38f57-e3c1-594f-6781-22113fe78d22} {Suffrage:Voter ID:server-09770b7f-cd60-a691-7d9c-d702b0609b28 Address:09770b7f-cd60-a691-7d9c-d702b0609b28} {Suffrage:Voter ID:server-19e4abba-dfde-6448-4f48-bf7641c8f371 Address:19e4abba-dfde-6448-4f48-bf7641c8f371} {Suffrage:Voter ID:server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 Address:293b25f1-82fb-cd6d-7127-8cdaa60a4e22}]" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.928Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3e38f57-e3c1-594f-6781-22113fe78d22 Address:c3e38f57-e3c1-594f-6781-22113fe78d22} {Suffrage:Voter ID:server-09770b7f-cd60-a691-7d9c-d702b0609b28 Address:09770b7f-cd60-a691-7d9c-d702b0609b28} {Suffrage:Voter ID:server-19e4abba-dfde-6448-4f48-bf7641c8f371 Address:19e4abba-dfde-6448-4f48-bf7641c8f371} {Suffrage:Voter ID:server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 Address:293b25f1-82fb-cd6d-7127-8cdaa60a4e22}]" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.928Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3e38f57-e3c1-594f-6781-22113fe78d22 Address:c3e38f57-e3c1-594f-6781-22113fe78d22} {Suffrage:Voter ID:server-09770b7f-cd60-a691-7d9c-d702b0609b28 Address:09770b7f-cd60-a691-7d9c-d702b0609b28} {Suffrage:Voter ID:server-19e4abba-dfde-6448-4f48-bf7641c8f371 Address:19e4abba-dfde-6448-4f48-bf7641c8f371} {Suffrage:Voter ID:server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 Address:293b25f1-82fb-cd6d-7127-8cdaa60a4e22}]" testLoggerAdapter verbose: server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22: 2025-02-09T16:52:14.928Z [INFO] server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3e38f57-e3c1-594f-6781-22113fe78d22 Address:c3e38f57-e3c1-594f-6781-22113fe78d22} {Suffrage:Voter ID:server-09770b7f-cd60-a691-7d9c-d702b0609b28 Address:09770b7f-cd60-a691-7d9c-d702b0609b28} {Suffrage:Voter ID:server-19e4abba-dfde-6448-4f48-bf7641c8f371 Address:19e4abba-dfde-6448-4f48-bf7641c8f371} {Suffrage:Voter ID:server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 Address:293b25f1-82fb-cd6d-7127-8cdaa60a4e22}]" testing.go:149: server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22: 2025-02-09T16:52:14.928Z [INFO] server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3e38f57-e3c1-594f-6781-22113fe78d22 Address:c3e38f57-e3c1-594f-6781-22113fe78d22} {Suffrage:Voter ID:server-09770b7f-cd60-a691-7d9c-d702b0609b28 Address:09770b7f-cd60-a691-7d9c-d702b0609b28} {Suffrage:Voter ID:server-19e4abba-dfde-6448-4f48-bf7641c8f371 Address:19e4abba-dfde-6448-4f48-bf7641c8f371} {Suffrage:Voter ID:server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 Address:293b25f1-82fb-cd6d-7127-8cdaa60a4e22}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:14.928Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:14.928Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c3e38f57-e3c1-594f-6781-22113fe78d22: 2025-02-09T16:52:14.928Z [INFO] server-c3e38f57-e3c1-594f-6781-22113fe78d22: entering follower state: follower="Node at c3e38f57-e3c1-594f-6781-22113fe78d22 [Follower]" leader= testing.go:149: server-c3e38f57-e3c1-594f-6781-22113fe78d22: 2025-02-09T16:52:14.928Z [INFO] server-c3e38f57-e3c1-594f-6781-22113fe78d22: entering follower state: follower="Node at c3e38f57-e3c1-594f-6781-22113fe78d22 [Follower]" leader= testLoggerAdapter verbose: server-09770b7f-cd60-a691-7d9c-d702b0609b28: 2025-02-09T16:52:14.928Z [INFO] server-09770b7f-cd60-a691-7d9c-d702b0609b28: entering follower state: follower="Node at 09770b7f-cd60-a691-7d9c-d702b0609b28 [Follower]" leader= testing.go:149: server-09770b7f-cd60-a691-7d9c-d702b0609b28: 2025-02-09T16:52:14.928Z [INFO] server-09770b7f-cd60-a691-7d9c-d702b0609b28: entering follower state: follower="Node at 09770b7f-cd60-a691-7d9c-d702b0609b28 [Follower]" leader= testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.928Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: entering follower state: follower="Node at 19e4abba-dfde-6448-4f48-bf7641c8f371 [Follower]" leader= testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.928Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: entering follower state: follower="Node at 19e4abba-dfde-6448-4f48-bf7641c8f371 [Follower]" leader= testLoggerAdapter verbose: server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22: 2025-02-09T16:52:14.928Z [INFO] server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22: entering follower state: follower="Node at 293b25f1-82fb-cd6d-7127-8cdaa60a4e22 [Follower]" leader= testing.go:149: server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22: 2025-02-09T16:52:14.928Z [INFO] server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22: entering follower state: follower="Node at 293b25f1-82fb-cd6d-7127-8cdaa60a4e22 [Follower]" leader= testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [WARN] server-19e4abba-dfde-6448-4f48-bf7641c8f371: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [WARN] server-19e4abba-dfde-6448-4f48-bf7641c8f371: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: entering candidate state: node="Node at 19e4abba-dfde-6448-4f48-bf7641c8f371 [Candidate]" term=2 testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: entering candidate state: node="Node at 19e4abba-dfde-6448-4f48-bf7641c8f371 [Candidate]" term=2 testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: election won: tally=3 testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: election won: tally=3 testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: entering leader state: leader="Node at 19e4abba-dfde-6448-4f48-bf7641c8f371 [Leader]" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: entering leader state: leader="Node at 19e4abba-dfde-6448-4f48-bf7641c8f371 [Leader]" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: added peer, starting replication: peer=server-c3e38f57-e3c1-594f-6781-22113fe78d22 testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: added peer, starting replication: peer=server-c3e38f57-e3c1-594f-6781-22113fe78d22 testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: added peer, starting replication: peer=server-09770b7f-cd60-a691-7d9c-d702b0609b28 testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: added peer, starting replication: peer=server-09770b7f-cd60-a691-7d9c-d702b0609b28 testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: added peer, starting replication: peer=server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: added peer, starting replication: peer=server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: pipelining replication: peer="{Voter server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 293b25f1-82fb-cd6d-7127-8cdaa60a4e22}" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: pipelining replication: peer="{Voter server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 293b25f1-82fb-cd6d-7127-8cdaa60a4e22}" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: pipelining replication: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: pipelining replication: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: pipelining replication: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:14.998Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: pipelining replication: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:15.104Z [INFO] stable state for Leader reached at 2025-02-09 16:52:14.998455643 +0000 UTC m=+9.328688331 (1 nodes), 70.167026ms from start of poll, 70.583158ms from cluster start. Timeout at 2025-02-09 16:52:15.104491827 +0000 UTC m=+9.434724513, 106.036182ms after stability testing.go:149: cluster: 2025-02-09T16:52:15.104Z [INFO] stable state for Leader reached at 2025-02-09 16:52:14.998455643 +0000 UTC m=+9.328688331 (1 nodes), 70.167026ms from start of poll, 70.583158ms from cluster start. Timeout at 2025-02-09 16:52:15.104491827 +0000 UTC m=+9.434724513, 106.036182ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:15.210Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:15.210Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:15.315Z [INFO] stable state for Follower reached at 2025-02-09 16:52:15.21017712 +0000 UTC m=+9.540409806 (3 nodes), 1.182µs from start of poll, 282.304633ms from cluster start. Timeout at 2025-02-09 16:52:15.315820469 +0000 UTC m=+9.646053156, 105.64335ms after stability testing.go:149: cluster: 2025-02-09T16:52:15.315Z [INFO] stable state for Follower reached at 2025-02-09 16:52:15.21017712 +0000 UTC m=+9.540409806 (3 nodes), 1.182µs from start of poll, 282.304633ms from cluster start. Timeout at 2025-02-09 16:52:15.315820469 +0000 UTC m=+9.646053156, 105.64335ms after stability testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.315Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: updating configuration: command=RemoveServer server-id=server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 server-addr= servers="[{Suffrage:Voter ID:server-c3e38f57-e3c1-594f-6781-22113fe78d22 Address:c3e38f57-e3c1-594f-6781-22113fe78d22} {Suffrage:Voter ID:server-09770b7f-cd60-a691-7d9c-d702b0609b28 Address:09770b7f-cd60-a691-7d9c-d702b0609b28} {Suffrage:Voter ID:server-19e4abba-dfde-6448-4f48-bf7641c8f371 Address:19e4abba-dfde-6448-4f48-bf7641c8f371}]" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.315Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: updating configuration: command=RemoveServer server-id=server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 server-addr= servers="[{Suffrage:Voter ID:server-c3e38f57-e3c1-594f-6781-22113fe78d22 Address:c3e38f57-e3c1-594f-6781-22113fe78d22} {Suffrage:Voter ID:server-09770b7f-cd60-a691-7d9c-d702b0609b28 Address:09770b7f-cd60-a691-7d9c-d702b0609b28} {Suffrage:Voter ID:server-19e4abba-dfde-6448-4f48-bf7641c8f371 Address:19e4abba-dfde-6448-4f48-bf7641c8f371}]" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: removed peer, stopping replication: peer=server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 last-index=3 testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: removed peer, stopping replication: peer=server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 last-index=3 testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to pipeline appendEntries: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" error="append pipeline closed" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to pipeline appendEntries: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" error="append pipeline closed" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: aborting pipeline replication: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: aborting pipeline replication: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to pipeline appendEntries: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" error="append pipeline closed" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to pipeline appendEntries: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" error="append pipeline closed" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: aborting pipeline replication: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: aborting pipeline replication: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: aborting pipeline replication: peer="{Voter server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 293b25f1-82fb-cd6d-7127-8cdaa60a4e22}" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: aborting pipeline replication: peer="{Voter server-293b25f1-82fb-cd6d-7127-8cdaa60a4e22 293b25f1-82fb-cd6d-7127-8cdaa60a4e22}" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=c3e38f57-e3c1-594f-6781-22113fe78d22 error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.316Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=c3e38f57-e3c1-594f-6781-22113fe78d22 error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.319Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=09770b7f-cd60-a691-7d9c-d702b0609b28 error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.319Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=09770b7f-cd60-a691-7d9c-d702b0609b28 error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.323Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.323Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.325Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.325Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.335Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=c3e38f57-e3c1-594f-6781-22113fe78d22 error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.335Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=c3e38f57-e3c1-594f-6781-22113fe78d22 error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.339Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=09770b7f-cd60-a691-7d9c-d702b0609b28 error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.339Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=09770b7f-cd60-a691-7d9c-d702b0609b28 error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.341Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.341Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.346Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.346Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.353Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=c3e38f57-e3c1-594f-6781-22113fe78d22 error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.353Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=c3e38f57-e3c1-594f-6781-22113fe78d22 error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.358Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=09770b7f-cd60-a691-7d9c-d702b0609b28 error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.358Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to heartbeat to: peer=09770b7f-cd60-a691-7d9c-d702b0609b28 error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.359Z [WARN] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to contact: server-id=server-c3e38f57-e3c1-594f-6781-22113fe78d22 time=50.439684ms testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.359Z [WARN] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to contact: server-id=server-c3e38f57-e3c1-594f-6781-22113fe78d22 time=50.439684ms testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.363Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.363Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-c3e38f57-e3c1-594f-6781-22113fe78d22 c3e38f57-e3c1-594f-6781-22113fe78d22}" error="failed to connect to peer: c3e38f57-e3c1-594f-6781-22113fe78d22" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.365Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.365Z [ERROR] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to appendEntries to: peer="{Voter server-09770b7f-cd60-a691-7d9c-d702b0609b28 09770b7f-cd60-a691-7d9c-d702b0609b28}" error="failed to connect to peer: 09770b7f-cd60-a691-7d9c-d702b0609b28" testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.370Z [WARN] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to contact: server-id=server-c3e38f57-e3c1-594f-6781-22113fe78d22 time=61.396625ms testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.370Z [WARN] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to contact: server-id=server-c3e38f57-e3c1-594f-6781-22113fe78d22 time=61.396625ms testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.370Z [WARN] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to contact: server-id=server-09770b7f-cd60-a691-7d9c-d702b0609b28 time=55.747352ms testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.370Z [WARN] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to contact: server-id=server-09770b7f-cd60-a691-7d9c-d702b0609b28 time=55.747352ms testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.370Z [WARN] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to contact quorum of nodes, stepping down testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.370Z [WARN] server-19e4abba-dfde-6448-4f48-bf7641c8f371: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.370Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: entering follower state: follower="Node at 19e4abba-dfde-6448-4f48-bf7641c8f371 [Follower]" leader= testing.go:149: server-19e4abba-dfde-6448-4f48-bf7641c8f371: 2025-02-09T16:52:15.370Z [INFO] server-19e4abba-dfde-6448-4f48-bf7641c8f371: entering follower state: follower="Node at 19e4abba-dfde-6448-4f48-bf7641c8f371 [Follower]" leader= --- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-3be3e906-9d8c-c3be-7a1e-e28ff224d947: 2025-02-09T16:52:15.371Z [INFO] server-3be3e906-9d8c-c3be-7a1e-e28ff224d947: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 Address:3be3e906-9d8c-c3be-7a1e-e28ff224d947} {Suffrage:Voter ID:server-49e70b91-7363-ff11-875c-c9d7155a2225 Address:49e70b91-7363-ff11-875c-c9d7155a2225} {Suffrage:Voter ID:server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 Address:c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}]" testing.go:149: server-3be3e906-9d8c-c3be-7a1e-e28ff224d947: 2025-02-09T16:52:15.371Z [INFO] server-3be3e906-9d8c-c3be-7a1e-e28ff224d947: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 Address:3be3e906-9d8c-c3be-7a1e-e28ff224d947} {Suffrage:Voter ID:server-49e70b91-7363-ff11-875c-c9d7155a2225 Address:49e70b91-7363-ff11-875c-c9d7155a2225} {Suffrage:Voter ID:server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 Address:c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}]" testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.371Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 Address:3be3e906-9d8c-c3be-7a1e-e28ff224d947} {Suffrage:Voter ID:server-49e70b91-7363-ff11-875c-c9d7155a2225 Address:49e70b91-7363-ff11-875c-c9d7155a2225} {Suffrage:Voter ID:server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 Address:c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}]" testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.371Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 Address:3be3e906-9d8c-c3be-7a1e-e28ff224d947} {Suffrage:Voter ID:server-49e70b91-7363-ff11-875c-c9d7155a2225 Address:49e70b91-7363-ff11-875c-c9d7155a2225} {Suffrage:Voter ID:server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 Address:c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}]" testLoggerAdapter verbose: server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4: 2025-02-09T16:52:15.371Z [INFO] server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 Address:3be3e906-9d8c-c3be-7a1e-e28ff224d947} {Suffrage:Voter ID:server-49e70b91-7363-ff11-875c-c9d7155a2225 Address:49e70b91-7363-ff11-875c-c9d7155a2225} {Suffrage:Voter ID:server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 Address:c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}]" testing.go:149: server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4: 2025-02-09T16:52:15.371Z [INFO] server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 Address:3be3e906-9d8c-c3be-7a1e-e28ff224d947} {Suffrage:Voter ID:server-49e70b91-7363-ff11-875c-c9d7155a2225 Address:49e70b91-7363-ff11-875c-c9d7155a2225} {Suffrage:Voter ID:server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 Address:c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:15.371Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:15.371Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-3be3e906-9d8c-c3be-7a1e-e28ff224d947: 2025-02-09T16:52:15.371Z [INFO] server-3be3e906-9d8c-c3be-7a1e-e28ff224d947: entering follower state: follower="Node at 3be3e906-9d8c-c3be-7a1e-e28ff224d947 [Follower]" leader= testing.go:149: server-3be3e906-9d8c-c3be-7a1e-e28ff224d947: 2025-02-09T16:52:15.371Z [INFO] server-3be3e906-9d8c-c3be-7a1e-e28ff224d947: entering follower state: follower="Node at 3be3e906-9d8c-c3be-7a1e-e28ff224d947 [Follower]" leader= testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.371Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: entering follower state: follower="Node at 49e70b91-7363-ff11-875c-c9d7155a2225 [Follower]" leader= testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.371Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: entering follower state: follower="Node at 49e70b91-7363-ff11-875c-c9d7155a2225 [Follower]" leader= testLoggerAdapter verbose: server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4: 2025-02-09T16:52:15.371Z [INFO] server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4: entering follower state: follower="Node at c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 [Follower]" leader= testing.go:149: server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4: 2025-02-09T16:52:15.371Z [INFO] server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4: entering follower state: follower="Node at c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 [Follower]" leader= testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.427Z [WARN] server-49e70b91-7363-ff11-875c-c9d7155a2225: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.427Z [WARN] server-49e70b91-7363-ff11-875c-c9d7155a2225: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.427Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: entering candidate state: node="Node at 49e70b91-7363-ff11-875c-c9d7155a2225 [Candidate]" term=2 testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.427Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: entering candidate state: node="Node at 49e70b91-7363-ff11-875c-c9d7155a2225 [Candidate]" term=2 testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.427Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: election won: tally=2 testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.427Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: election won: tally=2 testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.427Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: entering leader state: leader="Node at 49e70b91-7363-ff11-875c-c9d7155a2225 [Leader]" testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.427Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: entering leader state: leader="Node at 49e70b91-7363-ff11-875c-c9d7155a2225 [Leader]" testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.428Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: added peer, starting replication: peer=server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.428Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: added peer, starting replication: peer=server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.428Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: added peer, starting replication: peer=server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.428Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: added peer, starting replication: peer=server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.428Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: pipelining replication: peer="{Voter server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}" testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.428Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: pipelining replication: peer="{Voter server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}" testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.428Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: pipelining replication: peer="{Voter server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 3be3e906-9d8c-c3be-7a1e-e28ff224d947}" testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.428Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: pipelining replication: peer="{Voter server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 3be3e906-9d8c-c3be-7a1e-e28ff224d947}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:15.533Z [INFO] stable state for Leader reached at 2025-02-09 16:52:15.428278063 +0000 UTC m=+9.758510751 (1 nodes), 56.679621ms from start of poll, 57.043766ms from cluster start. Timeout at 2025-02-09 16:52:15.533340315 +0000 UTC m=+9.863573003, 105.062252ms after stability testing.go:149: cluster: 2025-02-09T16:52:15.533Z [INFO] stable state for Leader reached at 2025-02-09 16:52:15.428278063 +0000 UTC m=+9.758510751 (1 nodes), 56.679621ms from start of poll, 57.043766ms from cluster start. Timeout at 2025-02-09 16:52:15.533340315 +0000 UTC m=+9.863573003, 105.062252ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:15.533Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:15.533Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:15.638Z [INFO] stable state for Follower reached at 2025-02-09 16:52:15.533447091 +0000 UTC m=+9.863679777 (2 nodes), 962ns from start of poll, 162.212792ms from cluster start. Timeout at 2025-02-09 16:52:15.638483846 +0000 UTC m=+9.968716534, 105.036757ms after stability testing.go:149: cluster: 2025-02-09T16:52:15.638Z [INFO] stable state for Follower reached at 2025-02-09 16:52:15.533447091 +0000 UTC m=+9.863679777 (2 nodes), 962ns from start of poll, 162.212792ms from cluster start. Timeout at 2025-02-09 16:52:15.638483846 +0000 UTC m=+9.968716534, 105.036757ms after stability testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.638Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: updating configuration: command=AddStaging server-id=server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 server-addr=3be3e906-9d8c-c3be-7a1e-e28ff224d947 servers="[{Suffrage:Voter ID:server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 Address:3be3e906-9d8c-c3be-7a1e-e28ff224d947} {Suffrage:Voter ID:server-49e70b91-7363-ff11-875c-c9d7155a2225 Address:49e70b91-7363-ff11-875c-c9d7155a2225} {Suffrage:Voter ID:server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 Address:c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}]" testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.638Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: updating configuration: command=AddStaging server-id=server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 server-addr=3be3e906-9d8c-c3be-7a1e-e28ff224d947 servers="[{Suffrage:Voter ID:server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 Address:3be3e906-9d8c-c3be-7a1e-e28ff224d947} {Suffrage:Voter ID:server-49e70b91-7363-ff11-875c-c9d7155a2225 Address:49e70b91-7363-ff11-875c-c9d7155a2225} {Suffrage:Voter ID:server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 Address:c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}]" testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.638Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: aborting pipeline replication: peer="{Voter server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}" testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.638Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: aborting pipeline replication: peer="{Voter server-c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4 c47d2d9f-70ae-e21c-73fb-2f6ecd9860e4}" testLoggerAdapter verbose: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.638Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: aborting pipeline replication: peer="{Voter server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 3be3e906-9d8c-c3be-7a1e-e28ff224d947}" testing.go:149: server-49e70b91-7363-ff11-875c-c9d7155a2225: 2025-02-09T16:52:15.638Z [INFO] server-49e70b91-7363-ff11-875c-c9d7155a2225: aborting pipeline replication: peer="{Voter server-3be3e906-9d8c-c3be-7a1e-e28ff224d947 3be3e906-9d8c-c3be-7a1e-e28ff224d947}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.639Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 Address:7b56c9ae-b6dd-44ac-0348-b4ba927e47a6} {Suffrage:Voter ID:server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 Address:fe153dac-b9fd-a9f3-ae04-d729020bf2c3} {Suffrage:Voter ID:server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a Address:5f9c0d98-a09e-0d59-cb08-c743b9bf109a}]" testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.639Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 Address:7b56c9ae-b6dd-44ac-0348-b4ba927e47a6} {Suffrage:Voter ID:server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 Address:fe153dac-b9fd-a9f3-ae04-d729020bf2c3} {Suffrage:Voter ID:server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a Address:5f9c0d98-a09e-0d59-cb08-c743b9bf109a}]" testLoggerAdapter verbose: server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3: 2025-02-09T16:52:15.639Z [INFO] server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 Address:7b56c9ae-b6dd-44ac-0348-b4ba927e47a6} {Suffrage:Voter ID:server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 Address:fe153dac-b9fd-a9f3-ae04-d729020bf2c3} {Suffrage:Voter ID:server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a Address:5f9c0d98-a09e-0d59-cb08-c743b9bf109a}]" testing.go:149: server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3: 2025-02-09T16:52:15.639Z [INFO] server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 Address:7b56c9ae-b6dd-44ac-0348-b4ba927e47a6} {Suffrage:Voter ID:server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 Address:fe153dac-b9fd-a9f3-ae04-d729020bf2c3} {Suffrage:Voter ID:server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a Address:5f9c0d98-a09e-0d59-cb08-c743b9bf109a}]" testLoggerAdapter verbose: server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a: 2025-02-09T16:52:15.639Z [INFO] server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 Address:7b56c9ae-b6dd-44ac-0348-b4ba927e47a6} {Suffrage:Voter ID:server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 Address:fe153dac-b9fd-a9f3-ae04-d729020bf2c3} {Suffrage:Voter ID:server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a Address:5f9c0d98-a09e-0d59-cb08-c743b9bf109a}]" testing.go:149: server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a: 2025-02-09T16:52:15.639Z [INFO] server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 Address:7b56c9ae-b6dd-44ac-0348-b4ba927e47a6} {Suffrage:Voter ID:server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 Address:fe153dac-b9fd-a9f3-ae04-d729020bf2c3} {Suffrage:Voter ID:server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a Address:5f9c0d98-a09e-0d59-cb08-c743b9bf109a}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:15.639Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:15.639Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.639Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: entering follower state: follower="Node at 7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 [Follower]" leader= testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.639Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: entering follower state: follower="Node at 7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 [Follower]" leader= testLoggerAdapter verbose: server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3: 2025-02-09T16:52:15.639Z [INFO] server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3: entering follower state: follower="Node at fe153dac-b9fd-a9f3-ae04-d729020bf2c3 [Follower]" leader= testing.go:149: server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3: 2025-02-09T16:52:15.639Z [INFO] server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3: entering follower state: follower="Node at fe153dac-b9fd-a9f3-ae04-d729020bf2c3 [Follower]" leader= testLoggerAdapter verbose: server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a: 2025-02-09T16:52:15.639Z [INFO] server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a: entering follower state: follower="Node at 5f9c0d98-a09e-0d59-cb08-c743b9bf109a [Follower]" leader= testing.go:149: server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a: 2025-02-09T16:52:15.639Z [INFO] server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a: entering follower state: follower="Node at 5f9c0d98-a09e-0d59-cb08-c743b9bf109a [Follower]" leader= testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [WARN] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [WARN] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: entering candidate state: node="Node at 7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 [Candidate]" term=2 testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: entering candidate state: node="Node at 7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 [Candidate]" term=2 testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: election won: tally=2 testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: election won: tally=2 testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: entering leader state: leader="Node at 7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 [Leader]" testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: entering leader state: leader="Node at 7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 [Leader]" testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: added peer, starting replication: peer=server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: added peer, starting replication: peer=server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: added peer, starting replication: peer=server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.696Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: added peer, starting replication: peer=server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.697Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: pipelining replication: peer="{Voter server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a 5f9c0d98-a09e-0d59-cb08-c743b9bf109a}" testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.697Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: pipelining replication: peer="{Voter server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a 5f9c0d98-a09e-0d59-cb08-c743b9bf109a}" testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.697Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: pipelining replication: peer="{Voter server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 fe153dac-b9fd-a9f3-ae04-d729020bf2c3}" testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.697Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: pipelining replication: peer="{Voter server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 fe153dac-b9fd-a9f3-ae04-d729020bf2c3}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:15.802Z [INFO] stable state for Leader reached at 2025-02-09 16:52:15.69723948 +0000 UTC m=+10.027472168 (1 nodes), 57.542188ms from start of poll, 57.850002ms from cluster start. Timeout at 2025-02-09 16:52:15.802310342 +0000 UTC m=+10.132543029, 105.070861ms after stability testing.go:149: cluster: 2025-02-09T16:52:15.802Z [INFO] stable state for Leader reached at 2025-02-09 16:52:15.69723948 +0000 UTC m=+10.027472168 (1 nodes), 57.542188ms from start of poll, 57.850002ms from cluster start. Timeout at 2025-02-09 16:52:15.802310342 +0000 UTC m=+10.132543029, 105.070861ms after stability testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.802Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: updating configuration: command=RemoveServer server-id=d404cf5a-616f-bdc4-300c-ee60b5123120 server-addr= servers="[{Suffrage:Voter ID:server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 Address:7b56c9ae-b6dd-44ac-0348-b4ba927e47a6} {Suffrage:Voter ID:server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 Address:fe153dac-b9fd-a9f3-ae04-d729020bf2c3} {Suffrage:Voter ID:server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a Address:5f9c0d98-a09e-0d59-cb08-c743b9bf109a}]" testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.802Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: updating configuration: command=RemoveServer server-id=d404cf5a-616f-bdc4-300c-ee60b5123120 server-addr= servers="[{Suffrage:Voter ID:server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6 Address:7b56c9ae-b6dd-44ac-0348-b4ba927e47a6} {Suffrage:Voter ID:server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 Address:fe153dac-b9fd-a9f3-ae04-d729020bf2c3} {Suffrage:Voter ID:server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a Address:5f9c0d98-a09e-0d59-cb08-c743b9bf109a}]" testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.802Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: aborting pipeline replication: peer="{Voter server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a 5f9c0d98-a09e-0d59-cb08-c743b9bf109a}" testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.802Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: aborting pipeline replication: peer="{Voter server-5f9c0d98-a09e-0d59-cb08-c743b9bf109a 5f9c0d98-a09e-0d59-cb08-c743b9bf109a}" testLoggerAdapter verbose: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.802Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: aborting pipeline replication: peer="{Voter server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 fe153dac-b9fd-a9f3-ae04-d729020bf2c3}" testing.go:149: server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: 2025-02-09T16:52:15.802Z [INFO] server-7b56c9ae-b6dd-44ac-0348-b4ba927e47a6: aborting pipeline replication: peer="{Voter server-fe153dac-b9fd-a9f3-ae04-d729020bf2c3 fe153dac-b9fd-a9f3-ae04-d729020bf2c3}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.803Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8fa9bf50-64e2-9b92-32a3-b9224f66779f Address:8fa9bf50-64e2-9b92-32a3-b9224f66779f}]" testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.803Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8fa9bf50-64e2-9b92-32a3-b9224f66779f Address:8fa9bf50-64e2-9b92-32a3-b9224f66779f}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:15.803Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:15.803Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.803Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: entering follower state: follower="Node at 8fa9bf50-64e2-9b92-32a3-b9224f66779f [Follower]" leader= testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.803Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: entering follower state: follower="Node at 8fa9bf50-64e2-9b92-32a3-b9224f66779f [Follower]" leader= testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.894Z [WARN] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.894Z [WARN] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.894Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: entering candidate state: node="Node at 8fa9bf50-64e2-9b92-32a3-b9224f66779f [Candidate]" term=2 testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.894Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: entering candidate state: node="Node at 8fa9bf50-64e2-9b92-32a3-b9224f66779f [Candidate]" term=2 testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.894Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: election won: tally=1 testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.894Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: election won: tally=1 testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.895Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: entering leader state: leader="Node at 8fa9bf50-64e2-9b92-32a3-b9224f66779f [Leader]" testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:15.895Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: entering leader state: leader="Node at 8fa9bf50-64e2-9b92-32a3-b9224f66779f [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.000Z [INFO] stable state for Leader reached at 2025-02-09 16:52:15.895053455 +0000 UTC m=+10.225286142 (1 nodes), 91.826717ms from start of poll, 91.9664ms from cluster start. Timeout at 2025-02-09 16:52:16.00068491 +0000 UTC m=+10.330917615, 105.631473ms after stability testing.go:149: cluster: 2025-02-09T16:52:16.000Z [INFO] stable state for Leader reached at 2025-02-09 16:52:15.895053455 +0000 UTC m=+10.225286142 (1 nodes), 91.826717ms from start of poll, 91.9664ms from cluster start. Timeout at 2025-02-09 16:52:16.00068491 +0000 UTC m=+10.330917615, 105.631473ms after stability testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:16.001Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: starting snapshot up to: index=102 testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:16.001Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: starting snapshot up to: index=102 testing.go:153: 2025-02-09T16:52:16.001Z [INFO] creating new snapshot: path=/tmp/raft326523187/snapshots/2-102-1739119936001.tmp testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:16.002Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: compacting logs: from=1 to=92 testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:16.002Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:16.002Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: snapshot complete up to: index=102 testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:16.002Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: snapshot complete up to: index=102 testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:16.002Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: restored from snapshot: id=2-102-1739119936001 testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:16.002Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: restored from snapshot: id=2-102-1739119936001 testLoggerAdapter verbose: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:16.002Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8fa9bf50-64e2-9b92-32a3-b9224f66779f Address:8fa9bf50-64e2-9b92-32a3-b9224f66779f}]" testing.go:149: server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: 2025-02-09T16:52:16.002Z [INFO] server-8fa9bf50-64e2-9b92-32a3-b9224f66779f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8fa9bf50-64e2-9b92-32a3-b9224f66779f Address:8fa9bf50-64e2-9b92-32a3-b9224f66779f}]" --- PASS: TestRaft_SnapshotRestore (0.20s) === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.002Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67234341-ebc4-aabf-4cd3-de730dca4aaa Address:67234341-ebc4-aabf-4cd3-de730dca4aaa}]" testing.go:149: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.002Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67234341-ebc4-aabf-4cd3-de730dca4aaa Address:67234341-ebc4-aabf-4cd3-de730dca4aaa}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.002Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:16.002Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.002Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: entering follower state: follower="Node at 67234341-ebc4-aabf-4cd3-de730dca4aaa [Follower]" leader= testing.go:149: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.002Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: entering follower state: follower="Node at 67234341-ebc4-aabf-4cd3-de730dca4aaa [Follower]" leader= testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.060Z [WARN] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.060Z [WARN] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.060Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: entering candidate state: node="Node at 67234341-ebc4-aabf-4cd3-de730dca4aaa [Candidate]" term=2 testing.go:149: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.060Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: entering candidate state: node="Node at 67234341-ebc4-aabf-4cd3-de730dca4aaa [Candidate]" term=2 testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.060Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: election won: tally=1 testing.go:149: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.060Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: election won: tally=1 testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.060Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: entering leader state: leader="Node at 67234341-ebc4-aabf-4cd3-de730dca4aaa [Leader]" testing.go:149: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.060Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: entering leader state: leader="Node at 67234341-ebc4-aabf-4cd3-de730dca4aaa [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.166Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.060643101 +0000 UTC m=+10.390875789 (1 nodes), 57.922133ms from start of poll, 58.036825ms from cluster start. Timeout at 2025-02-09 16:52:16.166356145 +0000 UTC m=+10.496588833, 105.713044ms after stability testing.go:149: cluster: 2025-02-09T16:52:16.166Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.060643101 +0000 UTC m=+10.390875789 (1 nodes), 57.922133ms from start of poll, 58.036825ms from cluster start. Timeout at 2025-02-09 16:52:16.166356145 +0000 UTC m=+10.496588833, 105.713044ms after stability testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.167Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: starting snapshot up to: index=102 testing.go:149: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.167Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: starting snapshot up to: index=102 testing.go:153: 2025-02-09T16:52:16.167Z [INFO] creating new snapshot: path=/tmp/raft2285752075/snapshots/2-102-1739119936167.tmp testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.167Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: compacting logs: from=1 to=92 testing.go:149: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.167Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.167Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: snapshot complete up to: index=102 testing.go:149: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.167Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: snapshot complete up to: index=102 testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.167Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67234341-ebc4-aabf-4cd3-de730dca4aaa Address:67234341-ebc4-aabf-4cd3-de730dca4aaa}]" testing.go:149: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.167Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67234341-ebc4-aabf-4cd3-de730dca4aaa Address:67234341-ebc4-aabf-4cd3-de730dca4aaa}]" --- PASS: TestRaft_NoRestoreOnStart (0.17s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.168Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a76491d6-1dee-f059-61c7-0bc958d5ef63 Address:a76491d6-1dee-f059-61c7-0bc958d5ef63}]" testing.go:149: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.168Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a76491d6-1dee-f059-61c7-0bc958d5ef63 Address:a76491d6-1dee-f059-61c7-0bc958d5ef63}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.168Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:16.168Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.168Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: entering follower state: follower="Node at 67234341-ebc4-aabf-4cd3-de730dca4aaa [Follower]" leader= testLoggerAdapter verbose: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.168Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: entering follower state: follower="Node at a76491d6-1dee-f059-61c7-0bc958d5ef63 [Follower]" leader= testing.go:149: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.168Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: entering follower state: follower="Node at a76491d6-1dee-f059-61c7-0bc958d5ef63 [Follower]" leader= testLoggerAdapter verbose: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.224Z [WARN] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.224Z [WARN] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.225Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: entering candidate state: node="Node at a76491d6-1dee-f059-61c7-0bc958d5ef63 [Candidate]" term=2 testing.go:149: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.225Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: entering candidate state: node="Node at a76491d6-1dee-f059-61c7-0bc958d5ef63 [Candidate]" term=2 testLoggerAdapter verbose: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.225Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: election won: tally=1 testing.go:149: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.225Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: election won: tally=1 testLoggerAdapter verbose: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.225Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: entering leader state: leader="Node at a76491d6-1dee-f059-61c7-0bc958d5ef63 [Leader]" testing.go:149: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.225Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: entering leader state: leader="Node at a76491d6-1dee-f059-61c7-0bc958d5ef63 [Leader]" testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.258Z [WARN] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.258Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: entering candidate state: node="Node at 67234341-ebc4-aabf-4cd3-de730dca4aaa [Candidate]" term=3 testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.258Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: election won: tally=1 testLoggerAdapter verbose: server-67234341-ebc4-aabf-4cd3-de730dca4aaa: 2025-02-09T16:52:16.258Z [INFO] server-67234341-ebc4-aabf-4cd3-de730dca4aaa: entering leader state: leader="Node at 67234341-ebc4-aabf-4cd3-de730dca4aaa [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.330Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.225162553 +0000 UTC m=+10.555395241 (1 nodes), 57.045959ms from start of poll, 57.201733ms from cluster start. Timeout at 2025-02-09 16:52:16.330182933 +0000 UTC m=+10.660415621, 105.02038ms after stability testing.go:149: cluster: 2025-02-09T16:52:16.330Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.225162553 +0000 UTC m=+10.555395241 (1 nodes), 57.045959ms from start of poll, 57.201733ms from cluster start. Timeout at 2025-02-09 16:52:16.330182933 +0000 UTC m=+10.660415621, 105.02038ms after stability testLoggerAdapter verbose: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.330Z [ERROR] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.330Z [ERROR] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.330Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: starting snapshot up to: index=12 testing.go:149: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.330Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: starting snapshot up to: index=12 testing.go:153: 2025-02-09T16:52:16.330Z [INFO] creating new snapshot: path=/tmp/raft2356388670/snapshots/2-12-1739119936330.tmp testLoggerAdapter verbose: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.330Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: compacting logs: from=1 to=2 testing.go:149: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.330Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.330Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: snapshot complete up to: index=12 testing.go:149: server-a76491d6-1dee-f059-61c7-0bc958d5ef63: 2025-02-09T16:52:16.330Z [INFO] server-a76491d6-1dee-f059-61c7-0bc958d5ef63: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.16s) === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: 2025-02-09T16:52:16.331Z [INFO] server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 Address:823f772a-eb53-9ad3-91d8-2b96aab1ef25} {Suffrage:Voter ID:server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 Address:5b423e04-87c9-fe4d-07c4-6567a5ebee76} {Suffrage:Voter ID:server-1bec18b5-108d-b581-be1b-9ce0dfad75b3 Address:1bec18b5-108d-b581-be1b-9ce0dfad75b3}]" testing.go:149: server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: 2025-02-09T16:52:16.331Z [INFO] server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 Address:823f772a-eb53-9ad3-91d8-2b96aab1ef25} {Suffrage:Voter ID:server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 Address:5b423e04-87c9-fe4d-07c4-6567a5ebee76} {Suffrage:Voter ID:server-1bec18b5-108d-b581-be1b-9ce0dfad75b3 Address:1bec18b5-108d-b581-be1b-9ce0dfad75b3}]" testLoggerAdapter verbose: server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: 2025-02-09T16:52:16.331Z [INFO] server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 Address:823f772a-eb53-9ad3-91d8-2b96aab1ef25} {Suffrage:Voter ID:server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 Address:5b423e04-87c9-fe4d-07c4-6567a5ebee76} {Suffrage:Voter ID:server-1bec18b5-108d-b581-be1b-9ce0dfad75b3 Address:1bec18b5-108d-b581-be1b-9ce0dfad75b3}]" testing.go:149: server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: 2025-02-09T16:52:16.331Z [INFO] server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 Address:823f772a-eb53-9ad3-91d8-2b96aab1ef25} {Suffrage:Voter ID:server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 Address:5b423e04-87c9-fe4d-07c4-6567a5ebee76} {Suffrage:Voter ID:server-1bec18b5-108d-b581-be1b-9ce0dfad75b3 Address:1bec18b5-108d-b581-be1b-9ce0dfad75b3}]" testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.331Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 Address:823f772a-eb53-9ad3-91d8-2b96aab1ef25} {Suffrage:Voter ID:server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 Address:5b423e04-87c9-fe4d-07c4-6567a5ebee76} {Suffrage:Voter ID:server-1bec18b5-108d-b581-be1b-9ce0dfad75b3 Address:1bec18b5-108d-b581-be1b-9ce0dfad75b3}]" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.331Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 Address:823f772a-eb53-9ad3-91d8-2b96aab1ef25} {Suffrage:Voter ID:server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 Address:5b423e04-87c9-fe4d-07c4-6567a5ebee76} {Suffrage:Voter ID:server-1bec18b5-108d-b581-be1b-9ce0dfad75b3 Address:1bec18b5-108d-b581-be1b-9ce0dfad75b3}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.331Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:16.331Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: 2025-02-09T16:52:16.331Z [INFO] server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: entering follower state: follower="Node at 823f772a-eb53-9ad3-91d8-2b96aab1ef25 [Follower]" leader= testing.go:149: server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: 2025-02-09T16:52:16.331Z [INFO] server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: entering follower state: follower="Node at 823f772a-eb53-9ad3-91d8-2b96aab1ef25 [Follower]" leader= testLoggerAdapter verbose: server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: 2025-02-09T16:52:16.331Z [INFO] server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: entering follower state: follower="Node at 5b423e04-87c9-fe4d-07c4-6567a5ebee76 [Follower]" leader= testing.go:149: server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: 2025-02-09T16:52:16.331Z [INFO] server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: entering follower state: follower="Node at 5b423e04-87c9-fe4d-07c4-6567a5ebee76 [Follower]" leader= testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.331Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: entering follower state: follower="Node at 1bec18b5-108d-b581-be1b-9ce0dfad75b3 [Follower]" leader= testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.331Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: entering follower state: follower="Node at 1bec18b5-108d-b581-be1b-9ce0dfad75b3 [Follower]" leader= testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.404Z [WARN] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.404Z [WARN] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: entering candidate state: node="Node at 1bec18b5-108d-b581-be1b-9ce0dfad75b3 [Candidate]" term=2 testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: entering candidate state: node="Node at 1bec18b5-108d-b581-be1b-9ce0dfad75b3 [Candidate]" term=2 testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: election won: tally=2 testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: election won: tally=2 testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: entering leader state: leader="Node at 1bec18b5-108d-b581-be1b-9ce0dfad75b3 [Leader]" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: entering leader state: leader="Node at 1bec18b5-108d-b581-be1b-9ce0dfad75b3 [Leader]" testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: added peer, starting replication: peer=server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: added peer, starting replication: peer=server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: added peer, starting replication: peer=server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: added peer, starting replication: peer=server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: pipelining replication: peer="{Voter server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 5b423e04-87c9-fe4d-07c4-6567a5ebee76}" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: pipelining replication: peer="{Voter server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 5b423e04-87c9-fe4d-07c4-6567a5ebee76}" testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: pipelining replication: peer="{Voter server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 823f772a-eb53-9ad3-91d8-2b96aab1ef25}" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.405Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: pipelining replication: peer="{Voter server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 823f772a-eb53-9ad3-91d8-2b96aab1ef25}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.511Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.405433353 +0000 UTC m=+10.735666041 (1 nodes), 73.75506ms from start of poll, 74.107142ms from cluster start. Timeout at 2025-02-09 16:52:16.510978316 +0000 UTC m=+10.841211003, 105.544962ms after stability testing.go:149: cluster: 2025-02-09T16:52:16.511Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.405433353 +0000 UTC m=+10.735666041 (1 nodes), 73.75506ms from start of poll, 74.107142ms from cluster start. Timeout at 2025-02-09 16:52:16.510978316 +0000 UTC m=+10.841211003, 105.544962ms after stability testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.511Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: starting snapshot up to: index=12 testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.511Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: starting snapshot up to: index=12 testing.go:153: 2025-02-09T16:52:16.511Z [INFO] creating new snapshot: path=/tmp/raft3741098675/snapshots/2-12-1739119936511.tmp testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.512Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: snapshot complete up to: index=12 testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.512Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: snapshot complete up to: index=12 testing.go:153: 2025-02-09T16:52:16.514Z [INFO] creating new snapshot: path=/tmp/raft3741098675/snapshots/2-23-1739119936514.tmp testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.514Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: copied to local snapshot: bytes=71 testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.514Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.515Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: restored user snapshot: index=1 testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.515Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: restored user snapshot: index=1 testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.515Z [ERROR] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: failed to get log: index=23 error="log not found" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.515Z [ERROR] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.515Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: aborting pipeline replication: peer="{Voter server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 5b423e04-87c9-fe4d-07c4-6567a5ebee76}" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.515Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: aborting pipeline replication: peer="{Voter server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 5b423e04-87c9-fe4d-07c4-6567a5ebee76}" testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.515Z [ERROR] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: failed to get log: index=23 error="log not found" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.515Z [ERROR] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.515Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: aborting pipeline replication: peer="{Voter server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 823f772a-eb53-9ad3-91d8-2b96aab1ef25}" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.515Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: aborting pipeline replication: peer="{Voter server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 823f772a-eb53-9ad3-91d8-2b96aab1ef25}" testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.521Z [ERROR] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: failed to get log: index=23 error="log not found" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.521Z [ERROR] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:16.522Z [INFO] creating new snapshot: path=/tmp/raft1697805986/snapshots/2-23-1739119936522.tmp testLoggerAdapter verbose: server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: 2025-02-09T16:52:16.522Z [INFO] server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: copied to local snapshot: bytes=71 testing.go:149: server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: 2025-02-09T16:52:16.522Z [INFO] server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: 2025-02-09T16:52:16.522Z [INFO] server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: Installed remote snapshot testing.go:149: server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: 2025-02-09T16:52:16.522Z [INFO] server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: Installed remote snapshot testLoggerAdapter verbose: server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: 2025-02-09T16:52:16.522Z [WARN] server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: skipping application of old log: index=22 testing.go:149: server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: 2025-02-09T16:52:16.522Z [WARN] server-823f772a-eb53-9ad3-91d8-2b96aab1ef25: skipping application of old log: index=22 testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.522Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: pipelining replication: peer="{Voter server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 823f772a-eb53-9ad3-91d8-2b96aab1ef25}" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.522Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: pipelining replication: peer="{Voter server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 823f772a-eb53-9ad3-91d8-2b96aab1ef25}" testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.524Z [ERROR] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: failed to get log: index=23 error="log not found" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.524Z [ERROR] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:16.524Z [INFO] creating new snapshot: path=/tmp/raft2337416777/snapshots/2-23-1739119936524.tmp testLoggerAdapter verbose: server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: 2025-02-09T16:52:16.525Z [INFO] server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: copied to local snapshot: bytes=71 testing.go:149: server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: 2025-02-09T16:52:16.525Z [INFO] server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: 2025-02-09T16:52:16.525Z [INFO] server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: Installed remote snapshot testing.go:149: server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: 2025-02-09T16:52:16.525Z [INFO] server-5b423e04-87c9-fe4d-07c4-6567a5ebee76: Installed remote snapshot testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.525Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: pipelining replication: peer="{Voter server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 5b423e04-87c9-fe4d-07c4-6567a5ebee76}" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.525Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: pipelining replication: peer="{Voter server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 5b423e04-87c9-fe4d-07c4-6567a5ebee76}" testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.540Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: aborting pipeline replication: peer="{Voter server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 5b423e04-87c9-fe4d-07c4-6567a5ebee76}" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.540Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: aborting pipeline replication: peer="{Voter server-5b423e04-87c9-fe4d-07c4-6567a5ebee76 5b423e04-87c9-fe4d-07c4-6567a5ebee76}" testLoggerAdapter verbose: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.540Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: aborting pipeline replication: peer="{Voter server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 823f772a-eb53-9ad3-91d8-2b96aab1ef25}" testing.go:149: server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: 2025-02-09T16:52:16.540Z [INFO] server-1bec18b5-108d-b581-be1b-9ce0dfad75b3: aborting pipeline replication: peer="{Voter server-823f772a-eb53-9ad3-91d8-2b96aab1ef25 823f772a-eb53-9ad3-91d8-2b96aab1ef25}" testLoggerAdapter verbose: server-86ebc7ac-66bd-4408-8264-e0cd2136da97: 2025-02-09T16:52:16.540Z [INFO] server-86ebc7ac-66bd-4408-8264-e0cd2136da97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86ebc7ac-66bd-4408-8264-e0cd2136da97 Address:86ebc7ac-66bd-4408-8264-e0cd2136da97} {Suffrage:Voter ID:server-aff24f1d-7182-8979-02c3-856a74358352 Address:aff24f1d-7182-8979-02c3-856a74358352} {Suffrage:Voter ID:server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef Address:2f7f262d-e356-1e68-d5e2-9a64e844f2ef}]" testing.go:149: server-86ebc7ac-66bd-4408-8264-e0cd2136da97: 2025-02-09T16:52:16.540Z [INFO] server-86ebc7ac-66bd-4408-8264-e0cd2136da97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86ebc7ac-66bd-4408-8264-e0cd2136da97 Address:86ebc7ac-66bd-4408-8264-e0cd2136da97} {Suffrage:Voter ID:server-aff24f1d-7182-8979-02c3-856a74358352 Address:aff24f1d-7182-8979-02c3-856a74358352} {Suffrage:Voter ID:server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef Address:2f7f262d-e356-1e68-d5e2-9a64e844f2ef}]" testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.540Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86ebc7ac-66bd-4408-8264-e0cd2136da97 Address:86ebc7ac-66bd-4408-8264-e0cd2136da97} {Suffrage:Voter ID:server-aff24f1d-7182-8979-02c3-856a74358352 Address:aff24f1d-7182-8979-02c3-856a74358352} {Suffrage:Voter ID:server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef Address:2f7f262d-e356-1e68-d5e2-9a64e844f2ef}]" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.540Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86ebc7ac-66bd-4408-8264-e0cd2136da97 Address:86ebc7ac-66bd-4408-8264-e0cd2136da97} {Suffrage:Voter ID:server-aff24f1d-7182-8979-02c3-856a74358352 Address:aff24f1d-7182-8979-02c3-856a74358352} {Suffrage:Voter ID:server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef Address:2f7f262d-e356-1e68-d5e2-9a64e844f2ef}]" testLoggerAdapter verbose: server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: 2025-02-09T16:52:16.541Z [INFO] server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86ebc7ac-66bd-4408-8264-e0cd2136da97 Address:86ebc7ac-66bd-4408-8264-e0cd2136da97} {Suffrage:Voter ID:server-aff24f1d-7182-8979-02c3-856a74358352 Address:aff24f1d-7182-8979-02c3-856a74358352} {Suffrage:Voter ID:server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef Address:2f7f262d-e356-1e68-d5e2-9a64e844f2ef}]" testing.go:149: server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: 2025-02-09T16:52:16.541Z [INFO] server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86ebc7ac-66bd-4408-8264-e0cd2136da97 Address:86ebc7ac-66bd-4408-8264-e0cd2136da97} {Suffrage:Voter ID:server-aff24f1d-7182-8979-02c3-856a74358352 Address:aff24f1d-7182-8979-02c3-856a74358352} {Suffrage:Voter ID:server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef Address:2f7f262d-e356-1e68-d5e2-9a64e844f2ef}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.541Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:16.541Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-86ebc7ac-66bd-4408-8264-e0cd2136da97: 2025-02-09T16:52:16.541Z [INFO] server-86ebc7ac-66bd-4408-8264-e0cd2136da97: entering follower state: follower="Node at 86ebc7ac-66bd-4408-8264-e0cd2136da97 [Follower]" leader= testing.go:149: server-86ebc7ac-66bd-4408-8264-e0cd2136da97: 2025-02-09T16:52:16.541Z [INFO] server-86ebc7ac-66bd-4408-8264-e0cd2136da97: entering follower state: follower="Node at 86ebc7ac-66bd-4408-8264-e0cd2136da97 [Follower]" leader= testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.541Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: entering follower state: follower="Node at aff24f1d-7182-8979-02c3-856a74358352 [Follower]" leader= testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.541Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: entering follower state: follower="Node at aff24f1d-7182-8979-02c3-856a74358352 [Follower]" leader= testLoggerAdapter verbose: server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: 2025-02-09T16:52:16.541Z [INFO] server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: entering follower state: follower="Node at 2f7f262d-e356-1e68-d5e2-9a64e844f2ef [Follower]" leader= testing.go:149: server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: 2025-02-09T16:52:16.541Z [INFO] server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: entering follower state: follower="Node at 2f7f262d-e356-1e68-d5e2-9a64e844f2ef [Follower]" leader= testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [WARN] server-aff24f1d-7182-8979-02c3-856a74358352: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [WARN] server-aff24f1d-7182-8979-02c3-856a74358352: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: entering candidate state: node="Node at aff24f1d-7182-8979-02c3-856a74358352 [Candidate]" term=2 testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: entering candidate state: node="Node at aff24f1d-7182-8979-02c3-856a74358352 [Candidate]" term=2 testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: election won: tally=2 testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: election won: tally=2 testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: entering leader state: leader="Node at aff24f1d-7182-8979-02c3-856a74358352 [Leader]" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: entering leader state: leader="Node at aff24f1d-7182-8979-02c3-856a74358352 [Leader]" testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: added peer, starting replication: peer=server-86ebc7ac-66bd-4408-8264-e0cd2136da97 testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: added peer, starting replication: peer=server-86ebc7ac-66bd-4408-8264-e0cd2136da97 testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: added peer, starting replication: peer=server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: added peer, starting replication: peer=server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: pipelining replication: peer="{Voter server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef 2f7f262d-e356-1e68-d5e2-9a64e844f2ef}" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: pipelining replication: peer="{Voter server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef 2f7f262d-e356-1e68-d5e2-9a64e844f2ef}" testLoggerAdapter verbose: server-86ebc7ac-66bd-4408-8264-e0cd2136da97: 2025-02-09T16:52:16.595Z [WARN] server-86ebc7ac-66bd-4408-8264-e0cd2136da97: rejecting vote request since our last term is greater: candidate=aff24f1d-7182-8979-02c3-856a74358352 last-term=2 last-candidate-term=1 testing.go:149: server-86ebc7ac-66bd-4408-8264-e0cd2136da97: 2025-02-09T16:52:16.595Z [WARN] server-86ebc7ac-66bd-4408-8264-e0cd2136da97: rejecting vote request since our last term is greater: candidate=aff24f1d-7182-8979-02c3-856a74358352 last-term=2 last-candidate-term=1 testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: pipelining replication: peer="{Voter server-86ebc7ac-66bd-4408-8264-e0cd2136da97 86ebc7ac-66bd-4408-8264-e0cd2136da97}" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.595Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: pipelining replication: peer="{Voter server-86ebc7ac-66bd-4408-8264-e0cd2136da97 86ebc7ac-66bd-4408-8264-e0cd2136da97}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.701Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.595624035 +0000 UTC m=+10.925856723 (1 nodes), 54.489083ms from start of poll, 54.816494ms from cluster start. Timeout at 2025-02-09 16:52:16.701634006 +0000 UTC m=+11.031866700, 106.009977ms after stability testing.go:149: cluster: 2025-02-09T16:52:16.701Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.595624035 +0000 UTC m=+10.925856723 (1 nodes), 54.489083ms from start of poll, 54.816494ms from cluster start. Timeout at 2025-02-09 16:52:16.701634006 +0000 UTC m=+11.031866700, 106.009977ms after stability testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.702Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: starting snapshot up to: index=12 testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.702Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: starting snapshot up to: index=12 testing.go:153: 2025-02-09T16:52:16.702Z [INFO] creating new snapshot: path=/tmp/raft3823728204/snapshots/2-12-1739119936702.tmp testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.702Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: snapshot complete up to: index=12 testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.702Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: snapshot complete up to: index=12 testing.go:153: 2025-02-09T16:52:16.703Z [INFO] creating new snapshot: path=/tmp/raft3823728204/snapshots/2-23-1739119936703.tmp testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: copied to local snapshot: bytes=71 testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: restored user snapshot: index=1 testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: restored user snapshot: index=1 testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [ERROR] server-aff24f1d-7182-8979-02c3-856a74358352: failed to get log: index=23 error="log not found" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [ERROR] server-aff24f1d-7182-8979-02c3-856a74358352: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: aborting pipeline replication: peer="{Voter server-86ebc7ac-66bd-4408-8264-e0cd2136da97 86ebc7ac-66bd-4408-8264-e0cd2136da97}" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: aborting pipeline replication: peer="{Voter server-86ebc7ac-66bd-4408-8264-e0cd2136da97 86ebc7ac-66bd-4408-8264-e0cd2136da97}" testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [ERROR] server-aff24f1d-7182-8979-02c3-856a74358352: failed to get log: index=23 error="log not found" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [ERROR] server-aff24f1d-7182-8979-02c3-856a74358352: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: aborting pipeline replication: peer="{Voter server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef 2f7f262d-e356-1e68-d5e2-9a64e844f2ef}" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.703Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: aborting pipeline replication: peer="{Voter server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef 2f7f262d-e356-1e68-d5e2-9a64e844f2ef}" testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.709Z [ERROR] server-aff24f1d-7182-8979-02c3-856a74358352: failed to get log: index=23 error="log not found" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.709Z [ERROR] server-aff24f1d-7182-8979-02c3-856a74358352: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.709Z [ERROR] server-aff24f1d-7182-8979-02c3-856a74358352: failed to get log: index=23 error="log not found" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.709Z [ERROR] server-aff24f1d-7182-8979-02c3-856a74358352: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:16.709Z [INFO] creating new snapshot: path=/tmp/raft3402645027/snapshots/2-23-1739119936709.tmp testing.go:153: 2025-02-09T16:52:16.709Z [INFO] creating new snapshot: path=/tmp/raft1838337689/snapshots/2-23-1739119936709.tmp testLoggerAdapter verbose: server-86ebc7ac-66bd-4408-8264-e0cd2136da97: 2025-02-09T16:52:16.710Z [INFO] server-86ebc7ac-66bd-4408-8264-e0cd2136da97: copied to local snapshot: bytes=71 testing.go:149: server-86ebc7ac-66bd-4408-8264-e0cd2136da97: 2025-02-09T16:52:16.710Z [INFO] server-86ebc7ac-66bd-4408-8264-e0cd2136da97: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: 2025-02-09T16:52:16.710Z [INFO] server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: copied to local snapshot: bytes=71 testing.go:149: server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: 2025-02-09T16:52:16.710Z [INFO] server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: 2025-02-09T16:52:16.710Z [INFO] server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: Installed remote snapshot testing.go:149: server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: 2025-02-09T16:52:16.710Z [INFO] server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: Installed remote snapshot testLoggerAdapter verbose: server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: 2025-02-09T16:52:16.710Z [WARN] server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: skipping application of old log: index=22 testing.go:149: server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: 2025-02-09T16:52:16.710Z [WARN] server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef: skipping application of old log: index=22 testLoggerAdapter verbose: server-86ebc7ac-66bd-4408-8264-e0cd2136da97: 2025-02-09T16:52:16.710Z [INFO] server-86ebc7ac-66bd-4408-8264-e0cd2136da97: Installed remote snapshot testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.710Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: pipelining replication: peer="{Voter server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef 2f7f262d-e356-1e68-d5e2-9a64e844f2ef}" testing.go:149: server-86ebc7ac-66bd-4408-8264-e0cd2136da97: 2025-02-09T16:52:16.710Z [INFO] server-86ebc7ac-66bd-4408-8264-e0cd2136da97: Installed remote snapshot testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.710Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: pipelining replication: peer="{Voter server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef 2f7f262d-e356-1e68-d5e2-9a64e844f2ef}" testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.710Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: pipelining replication: peer="{Voter server-86ebc7ac-66bd-4408-8264-e0cd2136da97 86ebc7ac-66bd-4408-8264-e0cd2136da97}" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.710Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: pipelining replication: peer="{Voter server-86ebc7ac-66bd-4408-8264-e0cd2136da97 86ebc7ac-66bd-4408-8264-e0cd2136da97}" testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.721Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: aborting pipeline replication: peer="{Voter server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef 2f7f262d-e356-1e68-d5e2-9a64e844f2ef}" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.721Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: aborting pipeline replication: peer="{Voter server-2f7f262d-e356-1e68-d5e2-9a64e844f2ef 2f7f262d-e356-1e68-d5e2-9a64e844f2ef}" testLoggerAdapter verbose: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.721Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: aborting pipeline replication: peer="{Voter server-86ebc7ac-66bd-4408-8264-e0cd2136da97 86ebc7ac-66bd-4408-8264-e0cd2136da97}" testing.go:149: server-aff24f1d-7182-8979-02c3-856a74358352: 2025-02-09T16:52:16.721Z [INFO] server-aff24f1d-7182-8979-02c3-856a74358352: aborting pipeline replication: peer="{Voter server-86ebc7ac-66bd-4408-8264-e0cd2136da97 86ebc7ac-66bd-4408-8264-e0cd2136da97}" testLoggerAdapter verbose: server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: 2025-02-09T16:52:16.722Z [INFO] server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 Address:581f15ac-8f7d-e9a0-fb62-c8be0df97205} {Suffrage:Voter ID:server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd Address:b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd} {Suffrage:Voter ID:server-8a206b56-6874-9dd2-7ab4-d99682073983 Address:8a206b56-6874-9dd2-7ab4-d99682073983}]" testing.go:149: server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: 2025-02-09T16:52:16.722Z [INFO] server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 Address:581f15ac-8f7d-e9a0-fb62-c8be0df97205} {Suffrage:Voter ID:server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd Address:b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd} {Suffrage:Voter ID:server-8a206b56-6874-9dd2-7ab4-d99682073983 Address:8a206b56-6874-9dd2-7ab4-d99682073983}]" testLoggerAdapter verbose: server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: 2025-02-09T16:52:16.722Z [INFO] server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 Address:581f15ac-8f7d-e9a0-fb62-c8be0df97205} {Suffrage:Voter ID:server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd Address:b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd} {Suffrage:Voter ID:server-8a206b56-6874-9dd2-7ab4-d99682073983 Address:8a206b56-6874-9dd2-7ab4-d99682073983}]" testing.go:149: server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: 2025-02-09T16:52:16.722Z [INFO] server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 Address:581f15ac-8f7d-e9a0-fb62-c8be0df97205} {Suffrage:Voter ID:server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd Address:b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd} {Suffrage:Voter ID:server-8a206b56-6874-9dd2-7ab4-d99682073983 Address:8a206b56-6874-9dd2-7ab4-d99682073983}]" testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.722Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 Address:581f15ac-8f7d-e9a0-fb62-c8be0df97205} {Suffrage:Voter ID:server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd Address:b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd} {Suffrage:Voter ID:server-8a206b56-6874-9dd2-7ab4-d99682073983 Address:8a206b56-6874-9dd2-7ab4-d99682073983}]" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.722Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 Address:581f15ac-8f7d-e9a0-fb62-c8be0df97205} {Suffrage:Voter ID:server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd Address:b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd} {Suffrage:Voter ID:server-8a206b56-6874-9dd2-7ab4-d99682073983 Address:8a206b56-6874-9dd2-7ab4-d99682073983}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.722Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:16.722Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: 2025-02-09T16:52:16.722Z [INFO] server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: entering follower state: follower="Node at 581f15ac-8f7d-e9a0-fb62-c8be0df97205 [Follower]" leader= testing.go:149: server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: 2025-02-09T16:52:16.722Z [INFO] server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: entering follower state: follower="Node at 581f15ac-8f7d-e9a0-fb62-c8be0df97205 [Follower]" leader= testLoggerAdapter verbose: server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: 2025-02-09T16:52:16.722Z [INFO] server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: entering follower state: follower="Node at b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd [Follower]" leader= testing.go:149: server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: 2025-02-09T16:52:16.722Z [INFO] server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: entering follower state: follower="Node at b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd [Follower]" leader= testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.722Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: entering follower state: follower="Node at 8a206b56-6874-9dd2-7ab4-d99682073983 [Follower]" leader= testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.722Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: entering follower state: follower="Node at 8a206b56-6874-9dd2-7ab4-d99682073983 [Follower]" leader= testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [WARN] server-8a206b56-6874-9dd2-7ab4-d99682073983: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [WARN] server-8a206b56-6874-9dd2-7ab4-d99682073983: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: entering candidate state: node="Node at 8a206b56-6874-9dd2-7ab4-d99682073983 [Candidate]" term=2 testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: entering candidate state: node="Node at 8a206b56-6874-9dd2-7ab4-d99682073983 [Candidate]" term=2 testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: election won: tally=2 testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: election won: tally=2 testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: entering leader state: leader="Node at 8a206b56-6874-9dd2-7ab4-d99682073983 [Leader]" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: entering leader state: leader="Node at 8a206b56-6874-9dd2-7ab4-d99682073983 [Leader]" testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: added peer, starting replication: peer=server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: added peer, starting replication: peer=server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: added peer, starting replication: peer=server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: added peer, starting replication: peer=server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: pipelining replication: peer="{Voter server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd}" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: pipelining replication: peer="{Voter server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd}" testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: pipelining replication: peer="{Voter server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 581f15ac-8f7d-e9a0-fb62-c8be0df97205}" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.784Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: pipelining replication: peer="{Voter server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 581f15ac-8f7d-e9a0-fb62-c8be0df97205}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.890Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.784975141 +0000 UTC m=+11.115207829 (1 nodes), 62.126299ms from start of poll, 62.466268ms from cluster start. Timeout at 2025-02-09 16:52:16.890191083 +0000 UTC m=+11.220423771, 105.215942ms after stability testing.go:149: cluster: 2025-02-09T16:52:16.890Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.784975141 +0000 UTC m=+11.115207829 (1 nodes), 62.126299ms from start of poll, 62.466268ms from cluster start. Timeout at 2025-02-09 16:52:16.890191083 +0000 UTC m=+11.220423771, 105.215942ms after stability testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.890Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: starting snapshot up to: index=12 testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.890Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: starting snapshot up to: index=12 testing.go:153: 2025-02-09T16:52:16.890Z [INFO] creating new snapshot: path=/tmp/raft776709014/snapshots/2-12-1739119936890.tmp testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.891Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: snapshot complete up to: index=12 testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.891Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: snapshot complete up to: index=12 testing.go:153: 2025-02-09T16:52:16.891Z [INFO] creating new snapshot: path=/tmp/raft776709014/snapshots/2-23-1739119936891.tmp testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.891Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: copied to local snapshot: bytes=71 testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.891Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.893Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: restored user snapshot: index=1 testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.893Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: restored user snapshot: index=1 testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.893Z [ERROR] server-8a206b56-6874-9dd2-7ab4-d99682073983: failed to get log: index=23 error="log not found" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.893Z [ERROR] server-8a206b56-6874-9dd2-7ab4-d99682073983: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.893Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: aborting pipeline replication: peer="{Voter server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd}" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.893Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: aborting pipeline replication: peer="{Voter server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd}" testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.893Z [ERROR] server-8a206b56-6874-9dd2-7ab4-d99682073983: failed to get log: index=23 error="log not found" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.893Z [ERROR] server-8a206b56-6874-9dd2-7ab4-d99682073983: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.893Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: aborting pipeline replication: peer="{Voter server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 581f15ac-8f7d-e9a0-fb62-c8be0df97205}" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.893Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: aborting pipeline replication: peer="{Voter server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 581f15ac-8f7d-e9a0-fb62-c8be0df97205}" testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.902Z [ERROR] server-8a206b56-6874-9dd2-7ab4-d99682073983: failed to get log: index=23 error="log not found" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.902Z [ERROR] server-8a206b56-6874-9dd2-7ab4-d99682073983: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:16.903Z [INFO] creating new snapshot: path=/tmp/raft3275984814/snapshots/2-23-1739119936903.tmp testLoggerAdapter verbose: server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: 2025-02-09T16:52:16.903Z [INFO] server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: copied to local snapshot: bytes=71 testing.go:149: server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: 2025-02-09T16:52:16.903Z [INFO] server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: 2025-02-09T16:52:16.903Z [INFO] server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: Installed remote snapshot testing.go:149: server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: 2025-02-09T16:52:16.903Z [INFO] server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: Installed remote snapshot testLoggerAdapter verbose: server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: 2025-02-09T16:52:16.903Z [WARN] server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: skipping application of old log: index=22 testing.go:149: server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: 2025-02-09T16:52:16.903Z [WARN] server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd: skipping application of old log: index=22 testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.903Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: pipelining replication: peer="{Voter server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd}" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.903Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: pipelining replication: peer="{Voter server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd}" testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.903Z [ERROR] server-8a206b56-6874-9dd2-7ab4-d99682073983: failed to get log: index=23 error="log not found" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.903Z [ERROR] server-8a206b56-6874-9dd2-7ab4-d99682073983: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:16.903Z [INFO] creating new snapshot: path=/tmp/raft1450330207/snapshots/2-23-1739119936903.tmp testLoggerAdapter verbose: server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: 2025-02-09T16:52:16.904Z [INFO] server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: copied to local snapshot: bytes=71 testing.go:149: server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: 2025-02-09T16:52:16.904Z [INFO] server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: 2025-02-09T16:52:16.904Z [INFO] server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: Installed remote snapshot testing.go:149: server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: 2025-02-09T16:52:16.904Z [INFO] server-581f15ac-8f7d-e9a0-fb62-c8be0df97205: Installed remote snapshot testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.904Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: pipelining replication: peer="{Voter server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 581f15ac-8f7d-e9a0-fb62-c8be0df97205}" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.904Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: pipelining replication: peer="{Voter server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 581f15ac-8f7d-e9a0-fb62-c8be0df97205}" testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.921Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: aborting pipeline replication: peer="{Voter server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd}" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.921Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: aborting pipeline replication: peer="{Voter server-b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd b2491ab6-74e7-9dba-cdf1-46b1a85cc0dd}" testLoggerAdapter verbose: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.921Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: aborting pipeline replication: peer="{Voter server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 581f15ac-8f7d-e9a0-fb62-c8be0df97205}" testing.go:149: server-8a206b56-6874-9dd2-7ab4-d99682073983: 2025-02-09T16:52:16.921Z [INFO] server-8a206b56-6874-9dd2-7ab4-d99682073983: aborting pipeline replication: peer="{Voter server-581f15ac-8f7d-e9a0-fb62-c8be0df97205 581f15ac-8f7d-e9a0-fb62-c8be0df97205}" testLoggerAdapter verbose: server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: 2025-02-09T16:52:16.922Z [INFO] server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fee11124-1593-3e3d-f3ce-7797aacfb1dd Address:fee11124-1593-3e3d-f3ce-7797aacfb1dd} {Suffrage:Voter ID:server-96b0aae5-9061-8a70-9ec1-37c41708aca1 Address:96b0aae5-9061-8a70-9ec1-37c41708aca1} {Suffrage:Voter ID:server-76394e1b-1313-05db-990d-315968d6709d Address:76394e1b-1313-05db-990d-315968d6709d}]" testing.go:149: server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: 2025-02-09T16:52:16.922Z [INFO] server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fee11124-1593-3e3d-f3ce-7797aacfb1dd Address:fee11124-1593-3e3d-f3ce-7797aacfb1dd} {Suffrage:Voter ID:server-96b0aae5-9061-8a70-9ec1-37c41708aca1 Address:96b0aae5-9061-8a70-9ec1-37c41708aca1} {Suffrage:Voter ID:server-76394e1b-1313-05db-990d-315968d6709d Address:76394e1b-1313-05db-990d-315968d6709d}]" testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.922Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fee11124-1593-3e3d-f3ce-7797aacfb1dd Address:fee11124-1593-3e3d-f3ce-7797aacfb1dd} {Suffrage:Voter ID:server-96b0aae5-9061-8a70-9ec1-37c41708aca1 Address:96b0aae5-9061-8a70-9ec1-37c41708aca1} {Suffrage:Voter ID:server-76394e1b-1313-05db-990d-315968d6709d Address:76394e1b-1313-05db-990d-315968d6709d}]" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.922Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fee11124-1593-3e3d-f3ce-7797aacfb1dd Address:fee11124-1593-3e3d-f3ce-7797aacfb1dd} {Suffrage:Voter ID:server-96b0aae5-9061-8a70-9ec1-37c41708aca1 Address:96b0aae5-9061-8a70-9ec1-37c41708aca1} {Suffrage:Voter ID:server-76394e1b-1313-05db-990d-315968d6709d Address:76394e1b-1313-05db-990d-315968d6709d}]" testLoggerAdapter verbose: server-76394e1b-1313-05db-990d-315968d6709d: 2025-02-09T16:52:16.922Z [INFO] server-76394e1b-1313-05db-990d-315968d6709d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fee11124-1593-3e3d-f3ce-7797aacfb1dd Address:fee11124-1593-3e3d-f3ce-7797aacfb1dd} {Suffrage:Voter ID:server-96b0aae5-9061-8a70-9ec1-37c41708aca1 Address:96b0aae5-9061-8a70-9ec1-37c41708aca1} {Suffrage:Voter ID:server-76394e1b-1313-05db-990d-315968d6709d Address:76394e1b-1313-05db-990d-315968d6709d}]" testing.go:149: server-76394e1b-1313-05db-990d-315968d6709d: 2025-02-09T16:52:16.922Z [INFO] server-76394e1b-1313-05db-990d-315968d6709d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fee11124-1593-3e3d-f3ce-7797aacfb1dd Address:fee11124-1593-3e3d-f3ce-7797aacfb1dd} {Suffrage:Voter ID:server-96b0aae5-9061-8a70-9ec1-37c41708aca1 Address:96b0aae5-9061-8a70-9ec1-37c41708aca1} {Suffrage:Voter ID:server-76394e1b-1313-05db-990d-315968d6709d Address:76394e1b-1313-05db-990d-315968d6709d}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:16.922Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:16.922Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: 2025-02-09T16:52:16.922Z [INFO] server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: entering follower state: follower="Node at fee11124-1593-3e3d-f3ce-7797aacfb1dd [Follower]" leader= testing.go:149: server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: 2025-02-09T16:52:16.922Z [INFO] server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: entering follower state: follower="Node at fee11124-1593-3e3d-f3ce-7797aacfb1dd [Follower]" leader= testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.922Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: entering follower state: follower="Node at 96b0aae5-9061-8a70-9ec1-37c41708aca1 [Follower]" leader= testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.922Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: entering follower state: follower="Node at 96b0aae5-9061-8a70-9ec1-37c41708aca1 [Follower]" leader= testLoggerAdapter verbose: server-76394e1b-1313-05db-990d-315968d6709d: 2025-02-09T16:52:16.922Z [INFO] server-76394e1b-1313-05db-990d-315968d6709d: entering follower state: follower="Node at 76394e1b-1313-05db-990d-315968d6709d [Follower]" leader= testing.go:149: server-76394e1b-1313-05db-990d-315968d6709d: 2025-02-09T16:52:16.922Z [INFO] server-76394e1b-1313-05db-990d-315968d6709d: entering follower state: follower="Node at 76394e1b-1313-05db-990d-315968d6709d [Follower]" leader= testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.996Z [WARN] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.996Z [WARN] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: entering candidate state: node="Node at 96b0aae5-9061-8a70-9ec1-37c41708aca1 [Candidate]" term=2 testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: entering candidate state: node="Node at 96b0aae5-9061-8a70-9ec1-37c41708aca1 [Candidate]" term=2 testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: election won: tally=2 testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: election won: tally=2 testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: entering leader state: leader="Node at 96b0aae5-9061-8a70-9ec1-37c41708aca1 [Leader]" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: entering leader state: leader="Node at 96b0aae5-9061-8a70-9ec1-37c41708aca1 [Leader]" testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: added peer, starting replication: peer=server-fee11124-1593-3e3d-f3ce-7797aacfb1dd testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: added peer, starting replication: peer=server-fee11124-1593-3e3d-f3ce-7797aacfb1dd testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: added peer, starting replication: peer=server-76394e1b-1313-05db-990d-315968d6709d testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: added peer, starting replication: peer=server-76394e1b-1313-05db-990d-315968d6709d testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: pipelining replication: peer="{Voter server-76394e1b-1313-05db-990d-315968d6709d 76394e1b-1313-05db-990d-315968d6709d}" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: pipelining replication: peer="{Voter server-76394e1b-1313-05db-990d-315968d6709d 76394e1b-1313-05db-990d-315968d6709d}" testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: pipelining replication: peer="{Voter server-fee11124-1593-3e3d-f3ce-7797aacfb1dd fee11124-1593-3e3d-f3ce-7797aacfb1dd}" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:16.997Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: pipelining replication: peer="{Voter server-fee11124-1593-3e3d-f3ce-7797aacfb1dd fee11124-1593-3e3d-f3ce-7797aacfb1dd}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:17.103Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.997464867 +0000 UTC m=+11.327697555 (1 nodes), 75.135209ms from start of poll, 75.477832ms from cluster start. Timeout at 2025-02-09 16:52:17.103056321 +0000 UTC m=+11.433289009, 105.591454ms after stability testing.go:149: cluster: 2025-02-09T16:52:17.103Z [INFO] stable state for Leader reached at 2025-02-09 16:52:16.997464867 +0000 UTC m=+11.327697555 (1 nodes), 75.135209ms from start of poll, 75.477832ms from cluster start. Timeout at 2025-02-09 16:52:17.103056321 +0000 UTC m=+11.433289009, 105.591454ms after stability testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.103Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: starting snapshot up to: index=12 testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.103Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: starting snapshot up to: index=12 testing.go:153: 2025-02-09T16:52:17.103Z [INFO] creating new snapshot: path=/tmp/raft3855284576/snapshots/2-12-1739119937103.tmp testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.103Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: snapshot complete up to: index=12 testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.103Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: snapshot complete up to: index=12 testing.go:153: 2025-02-09T16:52:17.104Z [INFO] creating new snapshot: path=/tmp/raft3855284576/snapshots/2-113-1739119937104.tmp testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: copied to local snapshot: bytes=71 testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: restored user snapshot: index=1 testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: restored user snapshot: index=1 testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [ERROR] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: failed to get log: index=23 error="log not found" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [ERROR] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: aborting pipeline replication: peer="{Voter server-76394e1b-1313-05db-990d-315968d6709d 76394e1b-1313-05db-990d-315968d6709d}" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: aborting pipeline replication: peer="{Voter server-76394e1b-1313-05db-990d-315968d6709d 76394e1b-1313-05db-990d-315968d6709d}" testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [ERROR] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: failed to get log: index=23 error="log not found" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [ERROR] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: aborting pipeline replication: peer="{Voter server-fee11124-1593-3e3d-f3ce-7797aacfb1dd fee11124-1593-3e3d-f3ce-7797aacfb1dd}" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.104Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: aborting pipeline replication: peer="{Voter server-fee11124-1593-3e3d-f3ce-7797aacfb1dd fee11124-1593-3e3d-f3ce-7797aacfb1dd}" testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.113Z [ERROR] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: failed to get log: index=23 error="log not found" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.113Z [ERROR] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:17.114Z [INFO] creating new snapshot: path=/tmp/raft2192071279/snapshots/2-113-1739119937114.tmp testLoggerAdapter verbose: server-76394e1b-1313-05db-990d-315968d6709d: 2025-02-09T16:52:17.114Z [INFO] server-76394e1b-1313-05db-990d-315968d6709d: copied to local snapshot: bytes=71 testing.go:149: server-76394e1b-1313-05db-990d-315968d6709d: 2025-02-09T16:52:17.114Z [INFO] server-76394e1b-1313-05db-990d-315968d6709d: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-76394e1b-1313-05db-990d-315968d6709d: 2025-02-09T16:52:17.114Z [INFO] server-76394e1b-1313-05db-990d-315968d6709d: Installed remote snapshot testing.go:149: server-76394e1b-1313-05db-990d-315968d6709d: 2025-02-09T16:52:17.114Z [INFO] server-76394e1b-1313-05db-990d-315968d6709d: Installed remote snapshot testLoggerAdapter verbose: server-76394e1b-1313-05db-990d-315968d6709d: 2025-02-09T16:52:17.114Z [WARN] server-76394e1b-1313-05db-990d-315968d6709d: skipping application of old log: index=22 testing.go:149: server-76394e1b-1313-05db-990d-315968d6709d: 2025-02-09T16:52:17.114Z [WARN] server-76394e1b-1313-05db-990d-315968d6709d: skipping application of old log: index=22 testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.114Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: pipelining replication: peer="{Voter server-76394e1b-1313-05db-990d-315968d6709d 76394e1b-1313-05db-990d-315968d6709d}" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.114Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: pipelining replication: peer="{Voter server-76394e1b-1313-05db-990d-315968d6709d 76394e1b-1313-05db-990d-315968d6709d}" testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.114Z [ERROR] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: failed to get log: index=23 error="log not found" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.114Z [ERROR] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:17.114Z [INFO] creating new snapshot: path=/tmp/raft990092553/snapshots/2-113-1739119937114.tmp testLoggerAdapter verbose: server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: 2025-02-09T16:52:17.115Z [INFO] server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: copied to local snapshot: bytes=71 testing.go:149: server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: 2025-02-09T16:52:17.115Z [INFO] server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: 2025-02-09T16:52:17.115Z [INFO] server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: Installed remote snapshot testing.go:149: server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: 2025-02-09T16:52:17.115Z [INFO] server-fee11124-1593-3e3d-f3ce-7797aacfb1dd: Installed remote snapshot testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.115Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: pipelining replication: peer="{Voter server-fee11124-1593-3e3d-f3ce-7797aacfb1dd fee11124-1593-3e3d-f3ce-7797aacfb1dd}" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.115Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: pipelining replication: peer="{Voter server-fee11124-1593-3e3d-f3ce-7797aacfb1dd fee11124-1593-3e3d-f3ce-7797aacfb1dd}" testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.131Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: aborting pipeline replication: peer="{Voter server-76394e1b-1313-05db-990d-315968d6709d 76394e1b-1313-05db-990d-315968d6709d}" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.131Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: aborting pipeline replication: peer="{Voter server-76394e1b-1313-05db-990d-315968d6709d 76394e1b-1313-05db-990d-315968d6709d}" testLoggerAdapter verbose: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.131Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: aborting pipeline replication: peer="{Voter server-fee11124-1593-3e3d-f3ce-7797aacfb1dd fee11124-1593-3e3d-f3ce-7797aacfb1dd}" testing.go:149: server-96b0aae5-9061-8a70-9ec1-37c41708aca1: 2025-02-09T16:52:17.131Z [INFO] server-96b0aae5-9061-8a70-9ec1-37c41708aca1: aborting pipeline replication: peer="{Voter server-fee11124-1593-3e3d-f3ce-7797aacfb1dd fee11124-1593-3e3d-f3ce-7797aacfb1dd}" testLoggerAdapter verbose: server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: 2025-02-09T16:52:17.132Z [INFO] server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 Address:54982224-49d9-1f98-aeaa-e350b4d4ecd7} {Suffrage:Voter ID:server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d Address:7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d} {Suffrage:Voter ID:server-bf29de11-c0e7-07e1-7152-7d55102847c2 Address:bf29de11-c0e7-07e1-7152-7d55102847c2}]" testing.go:149: server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: 2025-02-09T16:52:17.132Z [INFO] server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 Address:54982224-49d9-1f98-aeaa-e350b4d4ecd7} {Suffrage:Voter ID:server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d Address:7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d} {Suffrage:Voter ID:server-bf29de11-c0e7-07e1-7152-7d55102847c2 Address:bf29de11-c0e7-07e1-7152-7d55102847c2}]" testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.132Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 Address:54982224-49d9-1f98-aeaa-e350b4d4ecd7} {Suffrage:Voter ID:server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d Address:7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d} {Suffrage:Voter ID:server-bf29de11-c0e7-07e1-7152-7d55102847c2 Address:bf29de11-c0e7-07e1-7152-7d55102847c2}]" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.132Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 Address:54982224-49d9-1f98-aeaa-e350b4d4ecd7} {Suffrage:Voter ID:server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d Address:7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d} {Suffrage:Voter ID:server-bf29de11-c0e7-07e1-7152-7d55102847c2 Address:bf29de11-c0e7-07e1-7152-7d55102847c2}]" testLoggerAdapter verbose: server-bf29de11-c0e7-07e1-7152-7d55102847c2: 2025-02-09T16:52:17.132Z [INFO] server-bf29de11-c0e7-07e1-7152-7d55102847c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 Address:54982224-49d9-1f98-aeaa-e350b4d4ecd7} {Suffrage:Voter ID:server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d Address:7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d} {Suffrage:Voter ID:server-bf29de11-c0e7-07e1-7152-7d55102847c2 Address:bf29de11-c0e7-07e1-7152-7d55102847c2}]" testing.go:149: server-bf29de11-c0e7-07e1-7152-7d55102847c2: 2025-02-09T16:52:17.132Z [INFO] server-bf29de11-c0e7-07e1-7152-7d55102847c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 Address:54982224-49d9-1f98-aeaa-e350b4d4ecd7} {Suffrage:Voter ID:server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d Address:7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d} {Suffrage:Voter ID:server-bf29de11-c0e7-07e1-7152-7d55102847c2 Address:bf29de11-c0e7-07e1-7152-7d55102847c2}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:17.132Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:17.132Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: 2025-02-09T16:52:17.132Z [INFO] server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: entering follower state: follower="Node at 54982224-49d9-1f98-aeaa-e350b4d4ecd7 [Follower]" leader= testing.go:149: server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: 2025-02-09T16:52:17.132Z [INFO] server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: entering follower state: follower="Node at 54982224-49d9-1f98-aeaa-e350b4d4ecd7 [Follower]" leader= testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.132Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: entering follower state: follower="Node at 7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d [Follower]" leader= testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.132Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: entering follower state: follower="Node at 7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d [Follower]" leader= testLoggerAdapter verbose: server-bf29de11-c0e7-07e1-7152-7d55102847c2: 2025-02-09T16:52:17.132Z [INFO] server-bf29de11-c0e7-07e1-7152-7d55102847c2: entering follower state: follower="Node at bf29de11-c0e7-07e1-7152-7d55102847c2 [Follower]" leader= testing.go:149: server-bf29de11-c0e7-07e1-7152-7d55102847c2: 2025-02-09T16:52:17.132Z [INFO] server-bf29de11-c0e7-07e1-7152-7d55102847c2: entering follower state: follower="Node at bf29de11-c0e7-07e1-7152-7d55102847c2 [Follower]" leader= testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [WARN] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [WARN] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: entering candidate state: node="Node at 7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d [Candidate]" term=2 testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: entering candidate state: node="Node at 7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d [Candidate]" term=2 testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: election won: tally=2 testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: election won: tally=2 testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: entering leader state: leader="Node at 7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d [Leader]" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: entering leader state: leader="Node at 7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d [Leader]" testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: added peer, starting replication: peer=server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: added peer, starting replication: peer=server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: added peer, starting replication: peer=server-bf29de11-c0e7-07e1-7152-7d55102847c2 testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: added peer, starting replication: peer=server-bf29de11-c0e7-07e1-7152-7d55102847c2 testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: pipelining replication: peer="{Voter server-bf29de11-c0e7-07e1-7152-7d55102847c2 bf29de11-c0e7-07e1-7152-7d55102847c2}" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: pipelining replication: peer="{Voter server-bf29de11-c0e7-07e1-7152-7d55102847c2 bf29de11-c0e7-07e1-7152-7d55102847c2}" testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: pipelining replication: peer="{Voter server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 54982224-49d9-1f98-aeaa-e350b4d4ecd7}" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.184Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: pipelining replication: peer="{Voter server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 54982224-49d9-1f98-aeaa-e350b4d4ecd7}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:17.290Z [INFO] stable state for Leader reached at 2025-02-09 16:52:17.184875701 +0000 UTC m=+11.515108389 (1 nodes), 52.103424ms from start of poll, 52.447801ms from cluster start. Timeout at 2025-02-09 16:52:17.290347538 +0000 UTC m=+11.620580226, 105.471837ms after stability testing.go:149: cluster: 2025-02-09T16:52:17.290Z [INFO] stable state for Leader reached at 2025-02-09 16:52:17.184875701 +0000 UTC m=+11.515108389 (1 nodes), 52.103424ms from start of poll, 52.447801ms from cluster start. Timeout at 2025-02-09 16:52:17.290347538 +0000 UTC m=+11.620580226, 105.471837ms after stability testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.290Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: starting snapshot up to: index=12 testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.290Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: starting snapshot up to: index=12 testing.go:153: 2025-02-09T16:52:17.290Z [INFO] creating new snapshot: path=/tmp/raft3163523233/snapshots/2-12-1739119937290.tmp testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.291Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: snapshot complete up to: index=12 testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.291Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: snapshot complete up to: index=12 testing.go:153: 2025-02-09T16:52:17.291Z [INFO] creating new snapshot: path=/tmp/raft3163523233/snapshots/2-1013-1739119937291.tmp testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.291Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: copied to local snapshot: bytes=71 testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.291Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.291Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: restored user snapshot: index=1 testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.291Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: restored user snapshot: index=1 testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.291Z [ERROR] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: failed to get log: index=23 error="log not found" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.291Z [ERROR] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.291Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: aborting pipeline replication: peer="{Voter server-bf29de11-c0e7-07e1-7152-7d55102847c2 bf29de11-c0e7-07e1-7152-7d55102847c2}" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.291Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: aborting pipeline replication: peer="{Voter server-bf29de11-c0e7-07e1-7152-7d55102847c2 bf29de11-c0e7-07e1-7152-7d55102847c2}" testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.292Z [ERROR] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: failed to get log: index=23 error="log not found" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.292Z [ERROR] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.292Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: aborting pipeline replication: peer="{Voter server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 54982224-49d9-1f98-aeaa-e350b4d4ecd7}" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.292Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: aborting pipeline replication: peer="{Voter server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 54982224-49d9-1f98-aeaa-e350b4d4ecd7}" testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.297Z [ERROR] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: failed to get log: index=23 error="log not found" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.297Z [ERROR] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:17.297Z [INFO] creating new snapshot: path=/tmp/raft3647749035/snapshots/2-1013-1739119937297.tmp testLoggerAdapter verbose: server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: 2025-02-09T16:52:17.298Z [INFO] server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: copied to local snapshot: bytes=71 testing.go:149: server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: 2025-02-09T16:52:17.298Z [INFO] server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: 2025-02-09T16:52:17.298Z [INFO] server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: Installed remote snapshot testing.go:149: server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: 2025-02-09T16:52:17.298Z [INFO] server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: Installed remote snapshot testLoggerAdapter verbose: server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: 2025-02-09T16:52:17.298Z [WARN] server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: skipping application of old log: index=22 testing.go:149: server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: 2025-02-09T16:52:17.298Z [WARN] server-54982224-49d9-1f98-aeaa-e350b4d4ecd7: skipping application of old log: index=22 testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.298Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: pipelining replication: peer="{Voter server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 54982224-49d9-1f98-aeaa-e350b4d4ecd7}" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.298Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: pipelining replication: peer="{Voter server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 54982224-49d9-1f98-aeaa-e350b4d4ecd7}" testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.300Z [ERROR] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: failed to get log: index=23 error="log not found" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.300Z [ERROR] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:17.301Z [INFO] creating new snapshot: path=/tmp/raft3597191872/snapshots/2-1013-1739119937301.tmp testLoggerAdapter verbose: server-bf29de11-c0e7-07e1-7152-7d55102847c2: 2025-02-09T16:52:17.301Z [INFO] server-bf29de11-c0e7-07e1-7152-7d55102847c2: copied to local snapshot: bytes=71 testing.go:149: server-bf29de11-c0e7-07e1-7152-7d55102847c2: 2025-02-09T16:52:17.301Z [INFO] server-bf29de11-c0e7-07e1-7152-7d55102847c2: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-bf29de11-c0e7-07e1-7152-7d55102847c2: 2025-02-09T16:52:17.301Z [INFO] server-bf29de11-c0e7-07e1-7152-7d55102847c2: Installed remote snapshot testing.go:149: server-bf29de11-c0e7-07e1-7152-7d55102847c2: 2025-02-09T16:52:17.301Z [INFO] server-bf29de11-c0e7-07e1-7152-7d55102847c2: Installed remote snapshot testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.301Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: pipelining replication: peer="{Voter server-bf29de11-c0e7-07e1-7152-7d55102847c2 bf29de11-c0e7-07e1-7152-7d55102847c2}" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.301Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: pipelining replication: peer="{Voter server-bf29de11-c0e7-07e1-7152-7d55102847c2 bf29de11-c0e7-07e1-7152-7d55102847c2}" testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.315Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: aborting pipeline replication: peer="{Voter server-bf29de11-c0e7-07e1-7152-7d55102847c2 bf29de11-c0e7-07e1-7152-7d55102847c2}" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.315Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: aborting pipeline replication: peer="{Voter server-bf29de11-c0e7-07e1-7152-7d55102847c2 bf29de11-c0e7-07e1-7152-7d55102847c2}" testLoggerAdapter verbose: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.315Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: aborting pipeline replication: peer="{Voter server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 54982224-49d9-1f98-aeaa-e350b4d4ecd7}" testing.go:149: server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: 2025-02-09T16:52:17.315Z [INFO] server-7207eff5-e4e0-aaa8-bf5d-487b2b8ea00d: aborting pipeline replication: peer="{Voter server-54982224-49d9-1f98-aeaa-e350b4d4ecd7 54982224-49d9-1f98-aeaa-e350b4d4ecd7}" testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.315Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca Address:2ef2ed47-5a01-1e5d-7708-db0f35e4ddca} {Suffrage:Voter ID:server-1edc9431-a467-130a-87a9-a35d599172b7 Address:1edc9431-a467-130a-87a9-a35d599172b7} {Suffrage:Voter ID:server-2188b6f3-9242-a299-91a8-681ae8da5c2c Address:2188b6f3-9242-a299-91a8-681ae8da5c2c}]" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.315Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca Address:2ef2ed47-5a01-1e5d-7708-db0f35e4ddca} {Suffrage:Voter ID:server-1edc9431-a467-130a-87a9-a35d599172b7 Address:1edc9431-a467-130a-87a9-a35d599172b7} {Suffrage:Voter ID:server-2188b6f3-9242-a299-91a8-681ae8da5c2c Address:2188b6f3-9242-a299-91a8-681ae8da5c2c}]" testLoggerAdapter verbose: server-1edc9431-a467-130a-87a9-a35d599172b7: 2025-02-09T16:52:17.315Z [INFO] server-1edc9431-a467-130a-87a9-a35d599172b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca Address:2ef2ed47-5a01-1e5d-7708-db0f35e4ddca} {Suffrage:Voter ID:server-1edc9431-a467-130a-87a9-a35d599172b7 Address:1edc9431-a467-130a-87a9-a35d599172b7} {Suffrage:Voter ID:server-2188b6f3-9242-a299-91a8-681ae8da5c2c Address:2188b6f3-9242-a299-91a8-681ae8da5c2c}]" testing.go:149: server-1edc9431-a467-130a-87a9-a35d599172b7: 2025-02-09T16:52:17.315Z [INFO] server-1edc9431-a467-130a-87a9-a35d599172b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca Address:2ef2ed47-5a01-1e5d-7708-db0f35e4ddca} {Suffrage:Voter ID:server-1edc9431-a467-130a-87a9-a35d599172b7 Address:1edc9431-a467-130a-87a9-a35d599172b7} {Suffrage:Voter ID:server-2188b6f3-9242-a299-91a8-681ae8da5c2c Address:2188b6f3-9242-a299-91a8-681ae8da5c2c}]" testLoggerAdapter verbose: server-2188b6f3-9242-a299-91a8-681ae8da5c2c: 2025-02-09T16:52:17.316Z [INFO] server-2188b6f3-9242-a299-91a8-681ae8da5c2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca Address:2ef2ed47-5a01-1e5d-7708-db0f35e4ddca} {Suffrage:Voter ID:server-1edc9431-a467-130a-87a9-a35d599172b7 Address:1edc9431-a467-130a-87a9-a35d599172b7} {Suffrage:Voter ID:server-2188b6f3-9242-a299-91a8-681ae8da5c2c Address:2188b6f3-9242-a299-91a8-681ae8da5c2c}]" testing.go:149: server-2188b6f3-9242-a299-91a8-681ae8da5c2c: 2025-02-09T16:52:17.316Z [INFO] server-2188b6f3-9242-a299-91a8-681ae8da5c2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca Address:2ef2ed47-5a01-1e5d-7708-db0f35e4ddca} {Suffrage:Voter ID:server-1edc9431-a467-130a-87a9-a35d599172b7 Address:1edc9431-a467-130a-87a9-a35d599172b7} {Suffrage:Voter ID:server-2188b6f3-9242-a299-91a8-681ae8da5c2c Address:2188b6f3-9242-a299-91a8-681ae8da5c2c}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:17.316Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:17.316Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.316Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: entering follower state: follower="Node at 2ef2ed47-5a01-1e5d-7708-db0f35e4ddca [Follower]" leader= testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.316Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: entering follower state: follower="Node at 2ef2ed47-5a01-1e5d-7708-db0f35e4ddca [Follower]" leader= testLoggerAdapter verbose: server-1edc9431-a467-130a-87a9-a35d599172b7: 2025-02-09T16:52:17.316Z [INFO] server-1edc9431-a467-130a-87a9-a35d599172b7: entering follower state: follower="Node at 1edc9431-a467-130a-87a9-a35d599172b7 [Follower]" leader= testing.go:149: server-1edc9431-a467-130a-87a9-a35d599172b7: 2025-02-09T16:52:17.316Z [INFO] server-1edc9431-a467-130a-87a9-a35d599172b7: entering follower state: follower="Node at 1edc9431-a467-130a-87a9-a35d599172b7 [Follower]" leader= testLoggerAdapter verbose: server-2188b6f3-9242-a299-91a8-681ae8da5c2c: 2025-02-09T16:52:17.316Z [INFO] server-2188b6f3-9242-a299-91a8-681ae8da5c2c: entering follower state: follower="Node at 2188b6f3-9242-a299-91a8-681ae8da5c2c [Follower]" leader= testing.go:149: server-2188b6f3-9242-a299-91a8-681ae8da5c2c: 2025-02-09T16:52:17.316Z [INFO] server-2188b6f3-9242-a299-91a8-681ae8da5c2c: entering follower state: follower="Node at 2188b6f3-9242-a299-91a8-681ae8da5c2c [Follower]" leader= testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.377Z [WARN] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.377Z [WARN] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.377Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: entering candidate state: node="Node at 2ef2ed47-5a01-1e5d-7708-db0f35e4ddca [Candidate]" term=2 testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.377Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: entering candidate state: node="Node at 2ef2ed47-5a01-1e5d-7708-db0f35e4ddca [Candidate]" term=2 testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.377Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: election won: tally=2 testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.377Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: election won: tally=2 testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.377Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: entering leader state: leader="Node at 2ef2ed47-5a01-1e5d-7708-db0f35e4ddca [Leader]" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.377Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: entering leader state: leader="Node at 2ef2ed47-5a01-1e5d-7708-db0f35e4ddca [Leader]" testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.378Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: added peer, starting replication: peer=server-1edc9431-a467-130a-87a9-a35d599172b7 testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.378Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: added peer, starting replication: peer=server-1edc9431-a467-130a-87a9-a35d599172b7 testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.378Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: added peer, starting replication: peer=server-2188b6f3-9242-a299-91a8-681ae8da5c2c testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.378Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: added peer, starting replication: peer=server-2188b6f3-9242-a299-91a8-681ae8da5c2c testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.378Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: pipelining replication: peer="{Voter server-2188b6f3-9242-a299-91a8-681ae8da5c2c 2188b6f3-9242-a299-91a8-681ae8da5c2c}" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.378Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: pipelining replication: peer="{Voter server-2188b6f3-9242-a299-91a8-681ae8da5c2c 2188b6f3-9242-a299-91a8-681ae8da5c2c}" testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.378Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: pipelining replication: peer="{Voter server-1edc9431-a467-130a-87a9-a35d599172b7 1edc9431-a467-130a-87a9-a35d599172b7}" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.378Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: pipelining replication: peer="{Voter server-1edc9431-a467-130a-87a9-a35d599172b7 1edc9431-a467-130a-87a9-a35d599172b7}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:17.483Z [INFO] stable state for Leader reached at 2025-02-09 16:52:17.37828519 +0000 UTC m=+11.708517878 (1 nodes), 62.164535ms from start of poll, 62.516419ms from cluster start. Timeout at 2025-02-09 16:52:17.483415198 +0000 UTC m=+11.813647887, 105.130009ms after stability testing.go:149: cluster: 2025-02-09T16:52:17.483Z [INFO] stable state for Leader reached at 2025-02-09 16:52:17.37828519 +0000 UTC m=+11.708517878 (1 nodes), 62.164535ms from start of poll, 62.516419ms from cluster start. Timeout at 2025-02-09 16:52:17.483415198 +0000 UTC m=+11.813647887, 105.130009ms after stability testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.485Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: starting snapshot up to: index=12 testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.485Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: starting snapshot up to: index=12 testing.go:153: 2025-02-09T16:52:17.485Z [INFO] creating new snapshot: path=/tmp/raft3617695562/snapshots/2-12-1739119937485.tmp testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.485Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: snapshot complete up to: index=12 testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.485Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: snapshot complete up to: index=12 testing.go:153: 2025-02-09T16:52:17.486Z [INFO] creating new snapshot: path=/tmp/raft3617695562/snapshots/2-10013-1739119937486.tmp testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.486Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: copied to local snapshot: bytes=71 testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.486Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.487Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: restored user snapshot: index=1 testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.487Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: restored user snapshot: index=1 testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.487Z [ERROR] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: failed to get log: index=23 error="log not found" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.487Z [ERROR] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.487Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: aborting pipeline replication: peer="{Voter server-2188b6f3-9242-a299-91a8-681ae8da5c2c 2188b6f3-9242-a299-91a8-681ae8da5c2c}" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.487Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: aborting pipeline replication: peer="{Voter server-2188b6f3-9242-a299-91a8-681ae8da5c2c 2188b6f3-9242-a299-91a8-681ae8da5c2c}" testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.487Z [ERROR] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: failed to get log: index=23 error="log not found" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.487Z [ERROR] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.487Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: aborting pipeline replication: peer="{Voter server-1edc9431-a467-130a-87a9-a35d599172b7 1edc9431-a467-130a-87a9-a35d599172b7}" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.487Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: aborting pipeline replication: peer="{Voter server-1edc9431-a467-130a-87a9-a35d599172b7 1edc9431-a467-130a-87a9-a35d599172b7}" testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.494Z [ERROR] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: failed to get log: index=23 error="log not found" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.494Z [ERROR] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:17.494Z [INFO] creating new snapshot: path=/tmp/raft3463572367/snapshots/2-10013-1739119937494.tmp testLoggerAdapter verbose: server-2188b6f3-9242-a299-91a8-681ae8da5c2c: 2025-02-09T16:52:17.494Z [INFO] server-2188b6f3-9242-a299-91a8-681ae8da5c2c: copied to local snapshot: bytes=71 testing.go:149: server-2188b6f3-9242-a299-91a8-681ae8da5c2c: 2025-02-09T16:52:17.494Z [INFO] server-2188b6f3-9242-a299-91a8-681ae8da5c2c: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-2188b6f3-9242-a299-91a8-681ae8da5c2c: 2025-02-09T16:52:17.494Z [INFO] server-2188b6f3-9242-a299-91a8-681ae8da5c2c: Installed remote snapshot testing.go:149: server-2188b6f3-9242-a299-91a8-681ae8da5c2c: 2025-02-09T16:52:17.494Z [INFO] server-2188b6f3-9242-a299-91a8-681ae8da5c2c: Installed remote snapshot testLoggerAdapter verbose: server-2188b6f3-9242-a299-91a8-681ae8da5c2c: 2025-02-09T16:52:17.494Z [WARN] server-2188b6f3-9242-a299-91a8-681ae8da5c2c: skipping application of old log: index=22 testing.go:149: server-2188b6f3-9242-a299-91a8-681ae8da5c2c: 2025-02-09T16:52:17.494Z [WARN] server-2188b6f3-9242-a299-91a8-681ae8da5c2c: skipping application of old log: index=22 testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.494Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: pipelining replication: peer="{Voter server-2188b6f3-9242-a299-91a8-681ae8da5c2c 2188b6f3-9242-a299-91a8-681ae8da5c2c}" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.494Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: pipelining replication: peer="{Voter server-2188b6f3-9242-a299-91a8-681ae8da5c2c 2188b6f3-9242-a299-91a8-681ae8da5c2c}" testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.497Z [ERROR] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: failed to get log: index=23 error="log not found" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.497Z [ERROR] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T16:52:17.497Z [INFO] creating new snapshot: path=/tmp/raft711378525/snapshots/2-10013-1739119937497.tmp testLoggerAdapter verbose: server-1edc9431-a467-130a-87a9-a35d599172b7: 2025-02-09T16:52:17.497Z [INFO] server-1edc9431-a467-130a-87a9-a35d599172b7: copied to local snapshot: bytes=71 testing.go:149: server-1edc9431-a467-130a-87a9-a35d599172b7: 2025-02-09T16:52:17.497Z [INFO] server-1edc9431-a467-130a-87a9-a35d599172b7: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-1edc9431-a467-130a-87a9-a35d599172b7: 2025-02-09T16:52:17.498Z [INFO] server-1edc9431-a467-130a-87a9-a35d599172b7: Installed remote snapshot testing.go:149: server-1edc9431-a467-130a-87a9-a35d599172b7: 2025-02-09T16:52:17.498Z [INFO] server-1edc9431-a467-130a-87a9-a35d599172b7: Installed remote snapshot testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.498Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: pipelining replication: peer="{Voter server-1edc9431-a467-130a-87a9-a35d599172b7 1edc9431-a467-130a-87a9-a35d599172b7}" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.498Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: pipelining replication: peer="{Voter server-1edc9431-a467-130a-87a9-a35d599172b7 1edc9431-a467-130a-87a9-a35d599172b7}" testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.517Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: aborting pipeline replication: peer="{Voter server-1edc9431-a467-130a-87a9-a35d599172b7 1edc9431-a467-130a-87a9-a35d599172b7}" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.517Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: aborting pipeline replication: peer="{Voter server-1edc9431-a467-130a-87a9-a35d599172b7 1edc9431-a467-130a-87a9-a35d599172b7}" testLoggerAdapter verbose: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.517Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: aborting pipeline replication: peer="{Voter server-2188b6f3-9242-a299-91a8-681ae8da5c2c 2188b6f3-9242-a299-91a8-681ae8da5c2c}" testing.go:149: server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: 2025-02-09T16:52:17.517Z [INFO] server-2ef2ed47-5a01-1e5d-7708-db0f35e4ddca: aborting pipeline replication: peer="{Voter server-2188b6f3-9242-a299-91a8-681ae8da5c2c 2188b6f3-9242-a299-91a8-681ae8da5c2c}" --- PASS: TestRaft_UserRestore (1.19s) === RUN TestRaft_SendSnapshotFollower testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.518Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e Address:6ea491d7-e147-ff1b-4f0b-4c1e2240f79e} {Suffrage:Voter ID:server-6267344d-d02b-48c7-a06d-968b1e08c1d4 Address:6267344d-d02b-48c7-a06d-968b1e08c1d4} {Suffrage:Voter ID:server-4772b2ac-3800-6684-aac5-b7eea888d169 Address:4772b2ac-3800-6684-aac5-b7eea888d169}]" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.518Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e Address:6ea491d7-e147-ff1b-4f0b-4c1e2240f79e} {Suffrage:Voter ID:server-6267344d-d02b-48c7-a06d-968b1e08c1d4 Address:6267344d-d02b-48c7-a06d-968b1e08c1d4} {Suffrage:Voter ID:server-4772b2ac-3800-6684-aac5-b7eea888d169 Address:4772b2ac-3800-6684-aac5-b7eea888d169}]" testLoggerAdapter verbose: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.518Z [INFO] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e Address:6ea491d7-e147-ff1b-4f0b-4c1e2240f79e} {Suffrage:Voter ID:server-6267344d-d02b-48c7-a06d-968b1e08c1d4 Address:6267344d-d02b-48c7-a06d-968b1e08c1d4} {Suffrage:Voter ID:server-4772b2ac-3800-6684-aac5-b7eea888d169 Address:4772b2ac-3800-6684-aac5-b7eea888d169}]" testing.go:149: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.518Z [INFO] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e Address:6ea491d7-e147-ff1b-4f0b-4c1e2240f79e} {Suffrage:Voter ID:server-6267344d-d02b-48c7-a06d-968b1e08c1d4 Address:6267344d-d02b-48c7-a06d-968b1e08c1d4} {Suffrage:Voter ID:server-4772b2ac-3800-6684-aac5-b7eea888d169 Address:4772b2ac-3800-6684-aac5-b7eea888d169}]" testLoggerAdapter verbose: server-4772b2ac-3800-6684-aac5-b7eea888d169: 2025-02-09T16:52:17.518Z [INFO] server-4772b2ac-3800-6684-aac5-b7eea888d169: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e Address:6ea491d7-e147-ff1b-4f0b-4c1e2240f79e} {Suffrage:Voter ID:server-6267344d-d02b-48c7-a06d-968b1e08c1d4 Address:6267344d-d02b-48c7-a06d-968b1e08c1d4} {Suffrage:Voter ID:server-4772b2ac-3800-6684-aac5-b7eea888d169 Address:4772b2ac-3800-6684-aac5-b7eea888d169}]" testing.go:149: server-4772b2ac-3800-6684-aac5-b7eea888d169: 2025-02-09T16:52:17.518Z [INFO] server-4772b2ac-3800-6684-aac5-b7eea888d169: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e Address:6ea491d7-e147-ff1b-4f0b-4c1e2240f79e} {Suffrage:Voter ID:server-6267344d-d02b-48c7-a06d-968b1e08c1d4 Address:6267344d-d02b-48c7-a06d-968b1e08c1d4} {Suffrage:Voter ID:server-4772b2ac-3800-6684-aac5-b7eea888d169 Address:4772b2ac-3800-6684-aac5-b7eea888d169}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:17.518Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:17.518Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.518Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: entering follower state: follower="Node at 6ea491d7-e147-ff1b-4f0b-4c1e2240f79e [Follower]" leader= testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.518Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: entering follower state: follower="Node at 6ea491d7-e147-ff1b-4f0b-4c1e2240f79e [Follower]" leader= testLoggerAdapter verbose: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.518Z [INFO] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: entering follower state: follower="Node at 6267344d-d02b-48c7-a06d-968b1e08c1d4 [Follower]" leader= testing.go:149: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.518Z [INFO] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: entering follower state: follower="Node at 6267344d-d02b-48c7-a06d-968b1e08c1d4 [Follower]" leader= testLoggerAdapter verbose: server-4772b2ac-3800-6684-aac5-b7eea888d169: 2025-02-09T16:52:17.518Z [INFO] server-4772b2ac-3800-6684-aac5-b7eea888d169: entering follower state: follower="Node at 4772b2ac-3800-6684-aac5-b7eea888d169 [Follower]" leader= testing.go:149: server-4772b2ac-3800-6684-aac5-b7eea888d169: 2025-02-09T16:52:17.518Z [INFO] server-4772b2ac-3800-6684-aac5-b7eea888d169: entering follower state: follower="Node at 4772b2ac-3800-6684-aac5-b7eea888d169 [Follower]" leader= testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [WARN] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [WARN] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: entering candidate state: node="Node at 6ea491d7-e147-ff1b-4f0b-4c1e2240f79e [Candidate]" term=2 testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: entering candidate state: node="Node at 6ea491d7-e147-ff1b-4f0b-4c1e2240f79e [Candidate]" term=2 testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: election won: tally=2 testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: election won: tally=2 testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: entering leader state: leader="Node at 6ea491d7-e147-ff1b-4f0b-4c1e2240f79e [Leader]" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: entering leader state: leader="Node at 6ea491d7-e147-ff1b-4f0b-4c1e2240f79e [Leader]" testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: added peer, starting replication: peer=server-6267344d-d02b-48c7-a06d-968b1e08c1d4 testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: added peer, starting replication: peer=server-6267344d-d02b-48c7-a06d-968b1e08c1d4 testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: added peer, starting replication: peer=server-4772b2ac-3800-6684-aac5-b7eea888d169 testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.603Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: added peer, starting replication: peer=server-4772b2ac-3800-6684-aac5-b7eea888d169 testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.604Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: pipelining replication: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.604Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: pipelining replication: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.604Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: pipelining replication: peer="{Voter server-4772b2ac-3800-6684-aac5-b7eea888d169 4772b2ac-3800-6684-aac5-b7eea888d169}" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.604Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: pipelining replication: peer="{Voter server-4772b2ac-3800-6684-aac5-b7eea888d169 4772b2ac-3800-6684-aac5-b7eea888d169}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:17.709Z [INFO] stable state for Follower reached at 2025-02-09 16:52:17.604003258 +0000 UTC m=+11.934235947 (2 nodes), 85.484355ms from start of poll, 85.83071ms from cluster start. Timeout at 2025-02-09 16:52:17.709235988 +0000 UTC m=+12.039468676, 105.232729ms after stability testing.go:149: cluster: 2025-02-09T16:52:17.709Z [INFO] stable state for Follower reached at 2025-02-09 16:52:17.604003258 +0000 UTC m=+11.934235947 (2 nodes), 85.484355ms from start of poll, 85.83071ms from cluster start. Timeout at 2025-02-09 16:52:17.709235988 +0000 UTC m=+12.039468676, 105.232729ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:17.709Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:17.709Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:17.815Z [INFO] stable state for Leader reached at 2025-02-09 16:52:17.709360738 +0000 UTC m=+12.039593423 (1 nodes), 828ns from start of poll, 191.188186ms from cluster start. Timeout at 2025-02-09 16:52:17.815380712 +0000 UTC m=+12.145613400, 106.019977ms after stability testing.go:149: cluster: 2025-02-09T16:52:17.815Z [INFO] stable state for Leader reached at 2025-02-09 16:52:17.709360738 +0000 UTC m=+12.039593423 (1 nodes), 828ns from start of poll, 191.188186ms from cluster start. Timeout at 2025-02-09 16:52:17.815380712 +0000 UTC m=+12.145613400, 106.019977ms after stability testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.815Z [ERROR] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: failed to pipeline appendEntries: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" error="append pipeline closed" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.815Z [ERROR] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: failed to pipeline appendEntries: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" error="append pipeline closed" testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.815Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: aborting pipeline replication: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.815Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: aborting pipeline replication: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.815Z [ERROR] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: failed to appendEntries to: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" error="failed to connect to peer: 6267344d-d02b-48c7-a06d-968b1e08c1d4" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.815Z [ERROR] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: failed to appendEntries to: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" error="failed to connect to peer: 6267344d-d02b-48c7-a06d-968b1e08c1d4" raft_test.go:1320: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.817Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: starting snapshot up to: index=102 testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.817Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: starting snapshot up to: index=102 testing.go:153: 2025-02-09T16:52:17.817Z [INFO] creating new snapshot: path=/tmp/raft833580904/snapshots/2-102-1739119937817.tmp testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.817Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: compacting logs: from=1 to=92 testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.817Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.817Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: snapshot complete up to: index=102 testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.817Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: snapshot complete up to: index=102 testLoggerAdapter verbose: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.817Z [ERROR] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.817Z [ERROR] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-4772b2ac-3800-6684-aac5-b7eea888d169: 2025-02-09T16:52:17.817Z [INFO] server-4772b2ac-3800-6684-aac5-b7eea888d169: starting snapshot up to: index=58 testing.go:149: server-4772b2ac-3800-6684-aac5-b7eea888d169: 2025-02-09T16:52:17.817Z [INFO] server-4772b2ac-3800-6684-aac5-b7eea888d169: starting snapshot up to: index=58 testing.go:153: 2025-02-09T16:52:17.817Z [INFO] creating new snapshot: path=/tmp/raft3121326612/snapshots/2-58-1739119937817.tmp testLoggerAdapter verbose: server-4772b2ac-3800-6684-aac5-b7eea888d169: 2025-02-09T16:52:17.817Z [INFO] server-4772b2ac-3800-6684-aac5-b7eea888d169: compacting logs: from=1 to=58 testing.go:149: server-4772b2ac-3800-6684-aac5-b7eea888d169: 2025-02-09T16:52:17.817Z [INFO] server-4772b2ac-3800-6684-aac5-b7eea888d169: compacting logs: from=1 to=58 testLoggerAdapter verbose: server-4772b2ac-3800-6684-aac5-b7eea888d169: 2025-02-09T16:52:17.817Z [INFO] server-4772b2ac-3800-6684-aac5-b7eea888d169: snapshot complete up to: index=58 testing.go:149: server-4772b2ac-3800-6684-aac5-b7eea888d169: 2025-02-09T16:52:17.817Z [INFO] server-4772b2ac-3800-6684-aac5-b7eea888d169: snapshot complete up to: index=58 testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.826Z [ERROR] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: failed to get log: index=2 error="log not found" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.826Z [ERROR] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: failed to get log: index=2 error="log not found" testing.go:153: 2025-02-09T16:52:17.827Z [INFO] creating new snapshot: path=/tmp/raft3835147658/snapshots/2-102-1739119937827.tmp testLoggerAdapter verbose: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.827Z [INFO] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: copied to local snapshot: bytes=693 testing.go:149: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.827Z [INFO] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.827Z [INFO] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: Installed remote snapshot testing.go:149: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.827Z [INFO] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: Installed remote snapshot testLoggerAdapter verbose: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.827Z [WARN] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: skipping application of old log: index=102 testing.go:149: server-6267344d-d02b-48c7-a06d-968b1e08c1d4: 2025-02-09T16:52:17.827Z [WARN] server-6267344d-d02b-48c7-a06d-968b1e08c1d4: skipping application of old log: index=102 testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.827Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: pipelining replication: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.827Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: pipelining replication: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.828Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: aborting pipeline replication: peer="{Voter server-4772b2ac-3800-6684-aac5-b7eea888d169 4772b2ac-3800-6684-aac5-b7eea888d169}" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.828Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: aborting pipeline replication: peer="{Voter server-4772b2ac-3800-6684-aac5-b7eea888d169 4772b2ac-3800-6684-aac5-b7eea888d169}" testLoggerAdapter verbose: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.828Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: aborting pipeline replication: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" testing.go:149: server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: 2025-02-09T16:52:17.828Z [INFO] server-6ea491d7-e147-ff1b-4f0b-4c1e2240f79e: aborting pipeline replication: peer="{Voter server-6267344d-d02b-48c7-a06d-968b1e08c1d4 6267344d-d02b-48c7-a06d-968b1e08c1d4}" --- PASS: TestRaft_SendSnapshotFollower (0.31s) === RUN TestRaft_SendSnapshotAndLogsFollower testLoggerAdapter verbose: server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: 2025-02-09T16:52:17.829Z [INFO] server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce Address:788eaf0f-d7c0-4bc4-2ebc-38371b1229ce} {Suffrage:Voter ID:server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 Address:bfbab2dc-4e32-aaca-d82e-6c30f5753420} {Suffrage:Voter ID:server-77b14b62-5745-b721-9731-fccc2b92448d Address:77b14b62-5745-b721-9731-fccc2b92448d}]" testing.go:149: server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: 2025-02-09T16:52:17.829Z [INFO] server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce Address:788eaf0f-d7c0-4bc4-2ebc-38371b1229ce} {Suffrage:Voter ID:server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 Address:bfbab2dc-4e32-aaca-d82e-6c30f5753420} {Suffrage:Voter ID:server-77b14b62-5745-b721-9731-fccc2b92448d Address:77b14b62-5745-b721-9731-fccc2b92448d}]" testLoggerAdapter verbose: server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: 2025-02-09T16:52:17.829Z [INFO] server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce Address:788eaf0f-d7c0-4bc4-2ebc-38371b1229ce} {Suffrage:Voter ID:server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 Address:bfbab2dc-4e32-aaca-d82e-6c30f5753420} {Suffrage:Voter ID:server-77b14b62-5745-b721-9731-fccc2b92448d Address:77b14b62-5745-b721-9731-fccc2b92448d}]" testing.go:149: server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: 2025-02-09T16:52:17.829Z [INFO] server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce Address:788eaf0f-d7c0-4bc4-2ebc-38371b1229ce} {Suffrage:Voter ID:server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 Address:bfbab2dc-4e32-aaca-d82e-6c30f5753420} {Suffrage:Voter ID:server-77b14b62-5745-b721-9731-fccc2b92448d Address:77b14b62-5745-b721-9731-fccc2b92448d}]" testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.829Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce Address:788eaf0f-d7c0-4bc4-2ebc-38371b1229ce} {Suffrage:Voter ID:server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 Address:bfbab2dc-4e32-aaca-d82e-6c30f5753420} {Suffrage:Voter ID:server-77b14b62-5745-b721-9731-fccc2b92448d Address:77b14b62-5745-b721-9731-fccc2b92448d}]" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.829Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce Address:788eaf0f-d7c0-4bc4-2ebc-38371b1229ce} {Suffrage:Voter ID:server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 Address:bfbab2dc-4e32-aaca-d82e-6c30f5753420} {Suffrage:Voter ID:server-77b14b62-5745-b721-9731-fccc2b92448d Address:77b14b62-5745-b721-9731-fccc2b92448d}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:17.829Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:17.829Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: 2025-02-09T16:52:17.829Z [INFO] server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: entering follower state: follower="Node at 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce [Follower]" leader= testing.go:149: server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: 2025-02-09T16:52:17.829Z [INFO] server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: entering follower state: follower="Node at 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce [Follower]" leader= testLoggerAdapter verbose: server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: 2025-02-09T16:52:17.829Z [INFO] server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: entering follower state: follower="Node at bfbab2dc-4e32-aaca-d82e-6c30f5753420 [Follower]" leader= testing.go:149: server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: 2025-02-09T16:52:17.829Z [INFO] server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: entering follower state: follower="Node at bfbab2dc-4e32-aaca-d82e-6c30f5753420 [Follower]" leader= testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.830Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: entering follower state: follower="Node at 77b14b62-5745-b721-9731-fccc2b92448d [Follower]" leader= testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.830Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: entering follower state: follower="Node at 77b14b62-5745-b721-9731-fccc2b92448d [Follower]" leader= testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [WARN] server-77b14b62-5745-b721-9731-fccc2b92448d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [WARN] server-77b14b62-5745-b721-9731-fccc2b92448d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: entering candidate state: node="Node at 77b14b62-5745-b721-9731-fccc2b92448d [Candidate]" term=2 testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: entering candidate state: node="Node at 77b14b62-5745-b721-9731-fccc2b92448d [Candidate]" term=2 testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: election won: tally=2 testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: election won: tally=2 testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: entering leader state: leader="Node at 77b14b62-5745-b721-9731-fccc2b92448d [Leader]" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: entering leader state: leader="Node at 77b14b62-5745-b721-9731-fccc2b92448d [Leader]" testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: added peer, starting replication: peer=server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: added peer, starting replication: peer=server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: added peer, starting replication: peer=server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: added peer, starting replication: peer=server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: pipelining replication: peer="{Voter server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 bfbab2dc-4e32-aaca-d82e-6c30f5753420}" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: pipelining replication: peer="{Voter server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 bfbab2dc-4e32-aaca-d82e-6c30f5753420}" testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: pipelining replication: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:17.901Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: pipelining replication: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.007Z [INFO] stable state for Follower reached at 2025-02-09 16:52:17.901943991 +0000 UTC m=+12.232176679 (2 nodes), 72.044533ms from start of poll, 72.399985ms from cluster start. Timeout at 2025-02-09 16:52:18.007606888 +0000 UTC m=+12.337839576, 105.662897ms after stability testing.go:149: cluster: 2025-02-09T16:52:18.007Z [INFO] stable state for Follower reached at 2025-02-09 16:52:17.901943991 +0000 UTC m=+12.232176679 (2 nodes), 72.044533ms from start of poll, 72.399985ms from cluster start. Timeout at 2025-02-09 16:52:18.007606888 +0000 UTC m=+12.337839576, 105.662897ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.007Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:18.007Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.113Z [INFO] stable state for Leader reached at 2025-02-09 16:52:18.007723312 +0000 UTC m=+12.337955998 (1 nodes), 766ns from start of poll, 178.179304ms from cluster start. Timeout at 2025-02-09 16:52:18.113534547 +0000 UTC m=+12.443767233, 105.811235ms after stability testing.go:149: cluster: 2025-02-09T16:52:18.113Z [INFO] stable state for Leader reached at 2025-02-09 16:52:18.007723312 +0000 UTC m=+12.337955998 (1 nodes), 766ns from start of poll, 178.179304ms from cluster start. Timeout at 2025-02-09 16:52:18.113534547 +0000 UTC m=+12.443767233, 105.811235ms after stability testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.113Z [ERROR] server-77b14b62-5745-b721-9731-fccc2b92448d: failed to pipeline appendEntries: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" error="append pipeline closed" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.113Z [ERROR] server-77b14b62-5745-b721-9731-fccc2b92448d: failed to pipeline appendEntries: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" error="append pipeline closed" testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.113Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: aborting pipeline replication: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.113Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: aborting pipeline replication: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.113Z [ERROR] server-77b14b62-5745-b721-9731-fccc2b92448d: failed to appendEntries to: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" error="failed to connect to peer: 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.113Z [ERROR] server-77b14b62-5745-b721-9731-fccc2b92448d: failed to appendEntries to: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" error="failed to connect to peer: 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce" raft_test.go:1362: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: 2025-02-09T16:52:18.115Z [ERROR] server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: 2025-02-09T16:52:18.115Z [ERROR] server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: 2025-02-09T16:52:18.115Z [INFO] server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: starting snapshot up to: index=98 testing.go:149: server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: 2025-02-09T16:52:18.115Z [INFO] server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: starting snapshot up to: index=98 testing.go:153: 2025-02-09T16:52:18.115Z [INFO] creating new snapshot: path=/tmp/raft777594996/snapshots/2-98-1739119938115.tmp testLoggerAdapter verbose: server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: 2025-02-09T16:52:18.115Z [INFO] server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: compacting logs: from=1 to=92 testing.go:149: server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: 2025-02-09T16:52:18.115Z [INFO] server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: 2025-02-09T16:52:18.115Z [INFO] server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: snapshot complete up to: index=98 testing.go:149: server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: 2025-02-09T16:52:18.115Z [INFO] server-bfbab2dc-4e32-aaca-d82e-6c30f5753420: snapshot complete up to: index=98 testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.115Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: starting snapshot up to: index=102 testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.115Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: starting snapshot up to: index=102 testing.go:153: 2025-02-09T16:52:18.115Z [INFO] creating new snapshot: path=/tmp/raft317753678/snapshots/2-102-1739119938115.tmp testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.116Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: compacting logs: from=1 to=92 testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.116Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.116Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: snapshot complete up to: index=102 testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.116Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: snapshot complete up to: index=102 testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.116Z [ERROR] server-77b14b62-5745-b721-9731-fccc2b92448d: failed to heartbeat to: peer=788eaf0f-d7c0-4bc4-2ebc-38371b1229ce error="failed to connect to peer: 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.116Z [ERROR] server-77b14b62-5745-b721-9731-fccc2b92448d: failed to heartbeat to: peer=788eaf0f-d7c0-4bc4-2ebc-38371b1229ce error="failed to connect to peer: 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce" raft_test.go:1383: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.123Z [ERROR] server-77b14b62-5745-b721-9731-fccc2b92448d: failed to get log: index=2 error="log not found" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.123Z [ERROR] server-77b14b62-5745-b721-9731-fccc2b92448d: failed to get log: index=2 error="log not found" testing.go:153: 2025-02-09T16:52:18.124Z [INFO] creating new snapshot: path=/tmp/raft1536789051/snapshots/2-102-1739119938124.tmp testLoggerAdapter verbose: server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: 2025-02-09T16:52:18.124Z [INFO] server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: copied to local snapshot: bytes=693 testing.go:149: server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: 2025-02-09T16:52:18.124Z [INFO] server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: 2025-02-09T16:52:18.124Z [INFO] server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: Installed remote snapshot testing.go:149: server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: 2025-02-09T16:52:18.124Z [INFO] server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce: Installed remote snapshot testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.124Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: pipelining replication: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.124Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: pipelining replication: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.130Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: aborting pipeline replication: peer="{Voter server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 bfbab2dc-4e32-aaca-d82e-6c30f5753420}" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.130Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: aborting pipeline replication: peer="{Voter server-bfbab2dc-4e32-aaca-d82e-6c30f5753420 bfbab2dc-4e32-aaca-d82e-6c30f5753420}" testLoggerAdapter verbose: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.130Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: aborting pipeline replication: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" testing.go:149: server-77b14b62-5745-b721-9731-fccc2b92448d: 2025-02-09T16:52:18.130Z [INFO] server-77b14b62-5745-b721-9731-fccc2b92448d: aborting pipeline replication: peer="{Voter server-788eaf0f-d7c0-4bc4-2ebc-38371b1229ce 788eaf0f-d7c0-4bc4-2ebc-38371b1229ce}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s) === RUN TestRaft_ReJoinFollower testLoggerAdapter verbose: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.131Z [INFO] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 Address:7d3dfcf7-db1c-c8ec-517d-b58b8ecca061} {Suffrage:Voter ID:server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d Address:a25c7164-4322-7d4c-2d44-76f5c6b8b64d} {Suffrage:Voter ID:server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 Address:6f58c0d3-2b07-0d32-0532-f7a56763bca9}]" testing.go:149: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.131Z [INFO] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 Address:7d3dfcf7-db1c-c8ec-517d-b58b8ecca061} {Suffrage:Voter ID:server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d Address:a25c7164-4322-7d4c-2d44-76f5c6b8b64d} {Suffrage:Voter ID:server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 Address:6f58c0d3-2b07-0d32-0532-f7a56763bca9}]" testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.131Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 Address:7d3dfcf7-db1c-c8ec-517d-b58b8ecca061} {Suffrage:Voter ID:server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d Address:a25c7164-4322-7d4c-2d44-76f5c6b8b64d} {Suffrage:Voter ID:server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 Address:6f58c0d3-2b07-0d32-0532-f7a56763bca9}]" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.131Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 Address:7d3dfcf7-db1c-c8ec-517d-b58b8ecca061} {Suffrage:Voter ID:server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d Address:a25c7164-4322-7d4c-2d44-76f5c6b8b64d} {Suffrage:Voter ID:server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 Address:6f58c0d3-2b07-0d32-0532-f7a56763bca9}]" testLoggerAdapter verbose: server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: 2025-02-09T16:52:18.131Z [INFO] server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 Address:7d3dfcf7-db1c-c8ec-517d-b58b8ecca061} {Suffrage:Voter ID:server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d Address:a25c7164-4322-7d4c-2d44-76f5c6b8b64d} {Suffrage:Voter ID:server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 Address:6f58c0d3-2b07-0d32-0532-f7a56763bca9}]" testing.go:149: server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: 2025-02-09T16:52:18.131Z [INFO] server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 Address:7d3dfcf7-db1c-c8ec-517d-b58b8ecca061} {Suffrage:Voter ID:server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d Address:a25c7164-4322-7d4c-2d44-76f5c6b8b64d} {Suffrage:Voter ID:server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 Address:6f58c0d3-2b07-0d32-0532-f7a56763bca9}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.131Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:18.131Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.131Z [INFO] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: entering follower state: follower="Node at 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 [Follower]" leader= testing.go:149: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.131Z [INFO] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: entering follower state: follower="Node at 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 [Follower]" leader= testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.131Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering follower state: follower="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Follower]" leader= testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.131Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering follower state: follower="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Follower]" leader= testLoggerAdapter verbose: server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: 2025-02-09T16:52:18.131Z [INFO] server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: entering follower state: follower="Node at 6f58c0d3-2b07-0d32-0532-f7a56763bca9 [Follower]" leader= testing.go:149: server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: 2025-02-09T16:52:18.131Z [INFO] server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: entering follower state: follower="Node at 6f58c0d3-2b07-0d32-0532-f7a56763bca9 [Follower]" leader= testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.207Z [WARN] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.207Z [WARN] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.207Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering candidate state: node="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Candidate]" term=2 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.207Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering candidate state: node="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Candidate]" term=2 testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: election won: tally=2 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: election won: tally=2 testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering leader state: leader="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Leader]" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering leader state: leader="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Leader]" testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: added peer, starting replication: peer=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: added peer, starting replication: peer=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: added peer, starting replication: peer=server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: added peer, starting replication: peer=server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: pipelining replication: peer="{Voter server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 6f58c0d3-2b07-0d32-0532-f7a56763bca9}" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: pipelining replication: peer="{Voter server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 6f58c0d3-2b07-0d32-0532-f7a56763bca9}" testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: pipelining replication: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.208Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: pipelining replication: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.313Z [INFO] stable state for Leader reached at 2025-02-09 16:52:18.208389713 +0000 UTC m=+12.538622401 (1 nodes), 76.894807ms from start of poll, 77.227731ms from cluster start. Timeout at 2025-02-09 16:52:18.313474802 +0000 UTC m=+12.643707488, 105.085087ms after stability testing.go:149: cluster: 2025-02-09T16:52:18.313Z [INFO] stable state for Leader reached at 2025-02-09 16:52:18.208389713 +0000 UTC m=+12.538622401 (1 nodes), 76.894807ms from start of poll, 77.227731ms from cluster start. Timeout at 2025-02-09 16:52:18.313474802 +0000 UTC m=+12.643707488, 105.085087ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.319Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:18.319Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.424Z [INFO] stable state for Follower reached at 2025-02-09 16:52:18.319080409 +0000 UTC m=+12.649313095 (2 nodes), 1.262µs from start of poll, 187.918425ms from cluster start. Timeout at 2025-02-09 16:52:18.424441751 +0000 UTC m=+12.754674439, 105.361344ms after stability testing.go:149: cluster: 2025-02-09T16:52:18.424Z [INFO] stable state for Follower reached at 2025-02-09 16:52:18.319080409 +0000 UTC m=+12.649313095 (2 nodes), 1.262µs from start of poll, 187.918425ms from cluster start. Timeout at 2025-02-09 16:52:18.424441751 +0000 UTC m=+12.754674439, 105.361344ms after stability testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.424Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: updating configuration: command=RemoveServer server-id=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 server-addr= servers="[{Suffrage:Voter ID:server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d Address:a25c7164-4322-7d4c-2d44-76f5c6b8b64d} {Suffrage:Voter ID:server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 Address:6f58c0d3-2b07-0d32-0532-f7a56763bca9}]" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.424Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: updating configuration: command=RemoveServer server-id=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 server-addr= servers="[{Suffrage:Voter ID:server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d Address:a25c7164-4322-7d4c-2d44-76f5c6b8b64d} {Suffrage:Voter ID:server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 Address:6f58c0d3-2b07-0d32-0532-f7a56763bca9}]" testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.424Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: removed peer, stopping replication: peer=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 last-index=3 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.424Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: removed peer, stopping replication: peer=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 last-index=3 testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.424Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: aborting pipeline replication: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.424Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: aborting pipeline replication: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" testLoggerAdapter verbose: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.512Z [WARN] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: heartbeat timeout reached, starting election: last-leader=a25c7164-4322-7d4c-2d44-76f5c6b8b64d testing.go:149: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.512Z [WARN] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: heartbeat timeout reached, starting election: last-leader=a25c7164-4322-7d4c-2d44-76f5c6b8b64d testLoggerAdapter verbose: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.512Z [INFO] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: entering candidate state: node="Node at 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 [Candidate]" term=3 testing.go:149: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.512Z [INFO] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: entering candidate state: node="Node at 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 [Candidate]" term=3 testLoggerAdapter verbose: server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: 2025-02-09T16:52:18.512Z [WARN] server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: rejecting vote request since we have a leader: from=7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 leader=a25c7164-4322-7d4c-2d44-76f5c6b8b64d testing.go:149: server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: 2025-02-09T16:52:18.512Z [WARN] server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: rejecting vote request since we have a leader: from=7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 leader=a25c7164-4322-7d4c-2d44-76f5c6b8b64d testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.512Z [WARN] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: rejecting vote request since we have a leader: from=7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 leader=a25c7164-4322-7d4c-2d44-76f5c6b8b64d testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.512Z [WARN] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: rejecting vote request since we have a leader: from=7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 leader=a25c7164-4322-7d4c-2d44-76f5c6b8b64d testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.530Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: updating configuration: command=AddStaging server-id=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 server-addr=7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 servers="[{Suffrage:Voter ID:server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d Address:a25c7164-4322-7d4c-2d44-76f5c6b8b64d} {Suffrage:Voter ID:server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 Address:6f58c0d3-2b07-0d32-0532-f7a56763bca9} {Suffrage:Voter ID:server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 Address:7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}]" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.530Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: updating configuration: command=AddStaging server-id=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 server-addr=7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 servers="[{Suffrage:Voter ID:server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d Address:a25c7164-4322-7d4c-2d44-76f5c6b8b64d} {Suffrage:Voter ID:server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 Address:6f58c0d3-2b07-0d32-0532-f7a56763bca9} {Suffrage:Voter ID:server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 Address:7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}]" testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.530Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: added peer, starting replication: peer=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.530Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: added peer, starting replication: peer=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.530Z [ERROR] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: peer has newer term, stopping replication: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.530Z [ERROR] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: peer has newer term, stopping replication: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.530Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering follower state: follower="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Follower]" leader= testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.530Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering follower state: follower="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Follower]" leader= testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.530Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:18.530Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.530Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: aborting pipeline replication: peer="{Voter server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 6f58c0d3-2b07-0d32-0532-f7a56763bca9}" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.530Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: aborting pipeline replication: peer="{Voter server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 6f58c0d3-2b07-0d32-0532-f7a56763bca9}" testLoggerAdapter verbose: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.588Z [WARN] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: Election timeout reached, restarting election testing.go:149: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.588Z [WARN] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: Election timeout reached, restarting election testLoggerAdapter verbose: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.588Z [INFO] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: entering candidate state: node="Node at 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 [Candidate]" term=4 testing.go:149: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.588Z [INFO] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: entering candidate state: node="Node at 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 [Candidate]" term=4 testLoggerAdapter verbose: server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: 2025-02-09T16:52:18.589Z [WARN] server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: rejecting vote request since we have a leader: from=7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 leader=a25c7164-4322-7d4c-2d44-76f5c6b8b64d testing.go:149: server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: 2025-02-09T16:52:18.589Z [WARN] server-6f58c0d3-2b07-0d32-0532-f7a56763bca9: rejecting vote request since we have a leader: from=7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 leader=a25c7164-4322-7d4c-2d44-76f5c6b8b64d testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.589Z [WARN] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: rejecting vote request since our last index is greater: candidate=7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 last-index=4 last-candidate-index=3 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.589Z [WARN] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: rejecting vote request since our last index is greater: candidate=7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 last-index=4 last-candidate-index=3 testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [WARN] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [WARN] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering candidate state: node="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Candidate]" term=5 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering candidate state: node="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Candidate]" term=5 testLoggerAdapter verbose: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.599Z [INFO] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: entering follower state: follower="Node at 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 [Follower]" leader= testing.go:149: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.599Z [INFO] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: entering follower state: follower="Node at 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 [Follower]" leader= testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: election won: tally=2 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: election won: tally=2 testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering leader state: leader="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Leader]" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: entering leader state: leader="Node at a25c7164-4322-7d4c-2d44-76f5c6b8b64d [Leader]" testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: added peer, starting replication: peer=server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: added peer, starting replication: peer=server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: added peer, starting replication: peer=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: added peer, starting replication: peer=server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 testLoggerAdapter verbose: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.599Z [WARN] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: failed to get previous log: previous-index=4 last-index=3 error="log not found" testing.go:149: server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: 2025-02-09T16:52:18.599Z [WARN] server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061: failed to get previous log: previous-index=4 last-index=3 error="log not found" testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [WARN] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: appendEntries rejected, sending older logs: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" next=4 testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [WARN] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: appendEntries rejected, sending older logs: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" next=4 testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: pipelining replication: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: pipelining replication: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: pipelining replication: peer="{Voter server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 6f58c0d3-2b07-0d32-0532-f7a56763bca9}" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.599Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: pipelining replication: peer="{Voter server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 6f58c0d3-2b07-0d32-0532-f7a56763bca9}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.705Z [INFO] stable state for Leader reached at 2025-02-09 16:52:18.599923161 +0000 UTC m=+12.930155849 (1 nodes), 69.347484ms from start of poll, 468.761179ms from cluster start. Timeout at 2025-02-09 16:52:18.70588223 +0000 UTC m=+13.036114918, 105.959069ms after stability testing.go:149: cluster: 2025-02-09T16:52:18.705Z [INFO] stable state for Leader reached at 2025-02-09 16:52:18.599923161 +0000 UTC m=+12.930155849 (1 nodes), 69.347484ms from start of poll, 468.761179ms from cluster start. Timeout at 2025-02-09 16:52:18.70588223 +0000 UTC m=+13.036114918, 105.959069ms after stability testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.706Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: aborting pipeline replication: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.706Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: aborting pipeline replication: peer="{Voter server-7d3dfcf7-db1c-c8ec-517d-b58b8ecca061 7d3dfcf7-db1c-c8ec-517d-b58b8ecca061}" testLoggerAdapter verbose: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.706Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: aborting pipeline replication: peer="{Voter server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 6f58c0d3-2b07-0d32-0532-f7a56763bca9}" testing.go:149: server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: 2025-02-09T16:52:18.706Z [INFO] server-a25c7164-4322-7d4c-2d44-76f5c6b8b64d: aborting pipeline replication: peer="{Voter server-6f58c0d3-2b07-0d32-0532-f7a56763bca9 6f58c0d3-2b07-0d32-0532-f7a56763bca9}" --- PASS: TestRaft_ReJoinFollower (0.58s) === RUN TestRaft_LeaderLeaseExpire testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.706Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c722b287-10c4-78d0-012e-bbd1685468ba Address:c722b287-10c4-78d0-012e-bbd1685468ba} {Suffrage:Voter ID:server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b Address:2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}]" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.706Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c722b287-10c4-78d0-012e-bbd1685468ba Address:c722b287-10c4-78d0-012e-bbd1685468ba} {Suffrage:Voter ID:server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b Address:2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}]" testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:18.706Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c722b287-10c4-78d0-012e-bbd1685468ba Address:c722b287-10c4-78d0-012e-bbd1685468ba} {Suffrage:Voter ID:server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b Address:2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}]" testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:18.706Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c722b287-10c4-78d0-012e-bbd1685468ba Address:c722b287-10c4-78d0-012e-bbd1685468ba} {Suffrage:Voter ID:server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b Address:2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.706Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:18.706Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.706Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering follower state: follower="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Follower]" leader= testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.706Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering follower state: follower="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Follower]" leader= testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:18.706Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering follower state: follower="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Follower]" leader= testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:18.706Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering follower state: follower="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Follower]" leader= testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering candidate state: node="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Candidate]" term=2 testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering candidate state: node="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Candidate]" term=2 testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: election won: tally=2 testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: election won: tally=2 testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering leader state: leader="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Leader]" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering leader state: leader="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Leader]" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: added peer, starting replication: peer=server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: added peer, starting replication: peer=server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: pipelining replication: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.776Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: pipelining replication: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.882Z [INFO] stable state for Leader reached at 2025-02-09 16:52:18.776609699 +0000 UTC m=+13.106842387 (1 nodes), 69.802838ms from start of poll, 70.050387ms from cluster start. Timeout at 2025-02-09 16:52:18.882144612 +0000 UTC m=+13.212377300, 105.534913ms after stability testing.go:149: cluster: 2025-02-09T16:52:18.882Z [INFO] stable state for Leader reached at 2025-02-09 16:52:18.776609699 +0000 UTC m=+13.106842387 (1 nodes), 69.802838ms from start of poll, 70.050387ms from cluster start. Timeout at 2025-02-09 16:52:18.882144612 +0000 UTC m=+13.212377300, 105.534913ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.887Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:18.887Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:18.993Z [INFO] stable state for Follower reached at 2025-02-09 16:52:18.887818349 +0000 UTC m=+13.218051035 (1 nodes), 709ns from start of poll, 181.259035ms from cluster start. Timeout at 2025-02-09 16:52:18.993686652 +0000 UTC m=+13.323919340, 105.868305ms after stability testing.go:149: cluster: 2025-02-09T16:52:18.993Z [INFO] stable state for Follower reached at 2025-02-09 16:52:18.887818349 +0000 UTC m=+13.218051035 (1 nodes), 709ns from start of poll, 181.259035ms from cluster start. Timeout at 2025-02-09 16:52:18.993686652 +0000 UTC m=+13.323919340, 105.868305ms after stability raft_test.go:1492: [INFO] Disconnecting Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Follower] testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.994Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to pipeline appendEntries: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="append pipeline closed" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.994Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to pipeline appendEntries: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="append pipeline closed" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.995Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: aborting pipeline replication: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.995Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: aborting pipeline replication: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.998Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to heartbeat to: peer=2e284aaf-108e-94ea-c7e0-08bbc1e65f5b error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:18.998Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to heartbeat to: peer=2e284aaf-108e-94ea-c7e0-08bbc1e65f5b error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.003Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to appendEntries to: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.003Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to appendEntries to: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.019Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to heartbeat to: peer=2e284aaf-108e-94ea-c7e0-08bbc1e65f5b error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.019Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to heartbeat to: peer=2e284aaf-108e-94ea-c7e0-08bbc1e65f5b error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.020Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to appendEntries to: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.020Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to appendEntries to: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.037Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to heartbeat to: peer=2e284aaf-108e-94ea-c7e0-08bbc1e65f5b error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.037Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to heartbeat to: peer=2e284aaf-108e-94ea-c7e0-08bbc1e65f5b error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.039Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to contact: server-id=server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b time=50.804467ms testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.039Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to contact: server-id=server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b time=50.804467ms testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.039Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to contact quorum of nodes, stepping down testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.039Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.039Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering follower state: follower="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Follower]" leader= testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.039Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering follower state: follower="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Follower]" leader= testLoggerAdapter verbose: cluster: 2025-02-09T16:52:19.039Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:19.039Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.040Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to appendEntries to: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.040Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to appendEntries to: peer="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.041Z [WARN] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: heartbeat timeout reached, starting election: last-leader=c722b287-10c4-78d0-012e-bbd1685468ba testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.041Z [WARN] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: heartbeat timeout reached, starting election: last-leader=c722b287-10c4-78d0-012e-bbd1685468ba testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.041Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering candidate state: node="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Candidate]" term=3 testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.041Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering candidate state: node="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Candidate]" term=3 testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.041Z [ERROR] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: failed to make requestVote RPC: target="{Voter server-c722b287-10c4-78d0-012e-bbd1685468ba c722b287-10c4-78d0-012e-bbd1685468ba}" error="failed to connect to peer: c722b287-10c4-78d0-012e-bbd1685468ba" testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.041Z [ERROR] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: failed to make requestVote RPC: target="{Voter server-c722b287-10c4-78d0-012e-bbd1685468ba c722b287-10c4-78d0-012e-bbd1685468ba}" error="failed to connect to peer: c722b287-10c4-78d0-012e-bbd1685468ba" testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.092Z [WARN] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: Election timeout reached, restarting election testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.092Z [WARN] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: Election timeout reached, restarting election testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.092Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering candidate state: node="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Candidate]" term=4 testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.092Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering candidate state: node="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Candidate]" term=4 testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.092Z [ERROR] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: failed to make requestVote RPC: target="{Voter server-c722b287-10c4-78d0-012e-bbd1685468ba c722b287-10c4-78d0-012e-bbd1685468ba}" error="failed to connect to peer: c722b287-10c4-78d0-012e-bbd1685468ba" testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.092Z [ERROR] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: failed to make requestVote RPC: target="{Voter server-c722b287-10c4-78d0-012e-bbd1685468ba c722b287-10c4-78d0-012e-bbd1685468ba}" error="failed to connect to peer: c722b287-10c4-78d0-012e-bbd1685468ba" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.105Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.105Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.105Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering candidate state: node="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Candidate]" term=3 testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.105Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering candidate state: node="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Candidate]" term=3 testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.105Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to make requestVote RPC: target="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.105Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to make requestVote RPC: target="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.152Z [WARN] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: Election timeout reached, restarting election testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.152Z [WARN] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: Election timeout reached, restarting election testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.152Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering candidate state: node="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Candidate]" term=5 testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.152Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering candidate state: node="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Candidate]" term=5 testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.152Z [ERROR] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: failed to make requestVote RPC: target="{Voter server-c722b287-10c4-78d0-012e-bbd1685468ba c722b287-10c4-78d0-012e-bbd1685468ba}" error="failed to connect to peer: c722b287-10c4-78d0-012e-bbd1685468ba" testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.152Z [ERROR] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: failed to make requestVote RPC: target="{Voter server-c722b287-10c4-78d0-012e-bbd1685468ba c722b287-10c4-78d0-012e-bbd1685468ba}" error="failed to connect to peer: c722b287-10c4-78d0-012e-bbd1685468ba" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.196Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: Election timeout reached, restarting election testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.196Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: Election timeout reached, restarting election testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.196Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering candidate state: node="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Candidate]" term=4 testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.196Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering candidate state: node="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Candidate]" term=4 testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.196Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to make requestVote RPC: target="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.196Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to make requestVote RPC: target="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:19.211Z [INFO] stable state for Leader reached at 2025-02-09 16:52:19.10593562 +0000 UTC m=+13.436168308 (0 nodes), 66.510475ms from start of poll, 399.376308ms from cluster start. Timeout at 2025-02-09 16:52:19.211887214 +0000 UTC m=+13.542119902, 105.951594ms after stability testing.go:149: cluster: 2025-02-09T16:52:19.211Z [INFO] stable state for Leader reached at 2025-02-09 16:52:19.10593562 +0000 UTC m=+13.436168308 (0 nodes), 66.510475ms from start of poll, 399.376308ms from cluster start. Timeout at 2025-02-09 16:52:19.211887214 +0000 UTC m=+13.542119902, 105.951594ms after stability testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.215Z [WARN] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: Election timeout reached, restarting election testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.215Z [WARN] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: Election timeout reached, restarting election testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.215Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering candidate state: node="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Candidate]" term=6 testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.215Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering candidate state: node="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Candidate]" term=6 testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.215Z [ERROR] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: failed to make requestVote RPC: target="{Voter server-c722b287-10c4-78d0-012e-bbd1685468ba c722b287-10c4-78d0-012e-bbd1685468ba}" error="failed to connect to peer: c722b287-10c4-78d0-012e-bbd1685468ba" testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.215Z [ERROR] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: failed to make requestVote RPC: target="{Voter server-c722b287-10c4-78d0-012e-bbd1685468ba c722b287-10c4-78d0-012e-bbd1685468ba}" error="failed to connect to peer: c722b287-10c4-78d0-012e-bbd1685468ba" testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.277Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: Election timeout reached, restarting election testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.277Z [WARN] server-c722b287-10c4-78d0-012e-bbd1685468ba: Election timeout reached, restarting election testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.277Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering candidate state: node="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Candidate]" term=5 testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.277Z [INFO] server-c722b287-10c4-78d0-012e-bbd1685468ba: entering candidate state: node="Node at c722b287-10c4-78d0-012e-bbd1685468ba [Candidate]" term=5 testLoggerAdapter verbose: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.277Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to make requestVote RPC: target="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testing.go:149: server-c722b287-10c4-78d0-012e-bbd1685468ba: 2025-02-09T16:52:19.277Z [ERROR] server-c722b287-10c4-78d0-012e-bbd1685468ba: failed to make requestVote RPC: target="{Voter server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b}" error="failed to connect to peer: 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b" testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.301Z [WARN] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: Election timeout reached, restarting election testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.301Z [WARN] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: Election timeout reached, restarting election testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.301Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering candidate state: node="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Candidate]" term=7 testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.301Z [INFO] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: entering candidate state: node="Node at 2e284aaf-108e-94ea-c7e0-08bbc1e65f5b [Candidate]" term=7 testLoggerAdapter verbose: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.301Z [ERROR] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: failed to make requestVote RPC: target="{Voter server-c722b287-10c4-78d0-012e-bbd1685468ba c722b287-10c4-78d0-012e-bbd1685468ba}" error="failed to connect to peer: c722b287-10c4-78d0-012e-bbd1685468ba" testing.go:149: server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: 2025-02-09T16:52:19.301Z [ERROR] server-2e284aaf-108e-94ea-c7e0-08bbc1e65f5b: failed to make requestVote RPC: target="{Voter server-c722b287-10c4-78d0-012e-bbd1685468ba c722b287-10c4-78d0-012e-bbd1685468ba}" error="failed to connect to peer: c722b287-10c4-78d0-012e-bbd1685468ba" --- PASS: TestRaft_LeaderLeaseExpire (0.61s) === RUN TestRaft_Barrier testLoggerAdapter verbose: server-a0d11a06-75af-130f-79f9-fdb8a55698cb: 2025-02-09T16:52:19.317Z [INFO] server-a0d11a06-75af-130f-79f9-fdb8a55698cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0d11a06-75af-130f-79f9-fdb8a55698cb Address:a0d11a06-75af-130f-79f9-fdb8a55698cb} {Suffrage:Voter ID:server-60f8640d-a16a-48bc-a470-158993ffe7cc Address:60f8640d-a16a-48bc-a470-158993ffe7cc} {Suffrage:Voter ID:server-b6323d63-d110-2b16-9070-c017a98118c0 Address:b6323d63-d110-2b16-9070-c017a98118c0}]" testing.go:149: server-a0d11a06-75af-130f-79f9-fdb8a55698cb: 2025-02-09T16:52:19.317Z [INFO] server-a0d11a06-75af-130f-79f9-fdb8a55698cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0d11a06-75af-130f-79f9-fdb8a55698cb Address:a0d11a06-75af-130f-79f9-fdb8a55698cb} {Suffrage:Voter ID:server-60f8640d-a16a-48bc-a470-158993ffe7cc Address:60f8640d-a16a-48bc-a470-158993ffe7cc} {Suffrage:Voter ID:server-b6323d63-d110-2b16-9070-c017a98118c0 Address:b6323d63-d110-2b16-9070-c017a98118c0}]" testLoggerAdapter verbose: server-60f8640d-a16a-48bc-a470-158993ffe7cc: 2025-02-09T16:52:19.317Z [INFO] server-60f8640d-a16a-48bc-a470-158993ffe7cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0d11a06-75af-130f-79f9-fdb8a55698cb Address:a0d11a06-75af-130f-79f9-fdb8a55698cb} {Suffrage:Voter ID:server-60f8640d-a16a-48bc-a470-158993ffe7cc Address:60f8640d-a16a-48bc-a470-158993ffe7cc} {Suffrage:Voter ID:server-b6323d63-d110-2b16-9070-c017a98118c0 Address:b6323d63-d110-2b16-9070-c017a98118c0}]" testing.go:149: server-60f8640d-a16a-48bc-a470-158993ffe7cc: 2025-02-09T16:52:19.317Z [INFO] server-60f8640d-a16a-48bc-a470-158993ffe7cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0d11a06-75af-130f-79f9-fdb8a55698cb Address:a0d11a06-75af-130f-79f9-fdb8a55698cb} {Suffrage:Voter ID:server-60f8640d-a16a-48bc-a470-158993ffe7cc Address:60f8640d-a16a-48bc-a470-158993ffe7cc} {Suffrage:Voter ID:server-b6323d63-d110-2b16-9070-c017a98118c0 Address:b6323d63-d110-2b16-9070-c017a98118c0}]" testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.317Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0d11a06-75af-130f-79f9-fdb8a55698cb Address:a0d11a06-75af-130f-79f9-fdb8a55698cb} {Suffrage:Voter ID:server-60f8640d-a16a-48bc-a470-158993ffe7cc Address:60f8640d-a16a-48bc-a470-158993ffe7cc} {Suffrage:Voter ID:server-b6323d63-d110-2b16-9070-c017a98118c0 Address:b6323d63-d110-2b16-9070-c017a98118c0}]" testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.317Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0d11a06-75af-130f-79f9-fdb8a55698cb Address:a0d11a06-75af-130f-79f9-fdb8a55698cb} {Suffrage:Voter ID:server-60f8640d-a16a-48bc-a470-158993ffe7cc Address:60f8640d-a16a-48bc-a470-158993ffe7cc} {Suffrage:Voter ID:server-b6323d63-d110-2b16-9070-c017a98118c0 Address:b6323d63-d110-2b16-9070-c017a98118c0}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:19.317Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:19.317Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a0d11a06-75af-130f-79f9-fdb8a55698cb: 2025-02-09T16:52:19.318Z [INFO] server-a0d11a06-75af-130f-79f9-fdb8a55698cb: entering follower state: follower="Node at a0d11a06-75af-130f-79f9-fdb8a55698cb [Follower]" leader= testing.go:149: server-a0d11a06-75af-130f-79f9-fdb8a55698cb: 2025-02-09T16:52:19.318Z [INFO] server-a0d11a06-75af-130f-79f9-fdb8a55698cb: entering follower state: follower="Node at a0d11a06-75af-130f-79f9-fdb8a55698cb [Follower]" leader= testLoggerAdapter verbose: server-60f8640d-a16a-48bc-a470-158993ffe7cc: 2025-02-09T16:52:19.318Z [INFO] server-60f8640d-a16a-48bc-a470-158993ffe7cc: entering follower state: follower="Node at 60f8640d-a16a-48bc-a470-158993ffe7cc [Follower]" leader= testing.go:149: server-60f8640d-a16a-48bc-a470-158993ffe7cc: 2025-02-09T16:52:19.318Z [INFO] server-60f8640d-a16a-48bc-a470-158993ffe7cc: entering follower state: follower="Node at 60f8640d-a16a-48bc-a470-158993ffe7cc [Follower]" leader= testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.318Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: entering follower state: follower="Node at b6323d63-d110-2b16-9070-c017a98118c0 [Follower]" leader= testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.318Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: entering follower state: follower="Node at b6323d63-d110-2b16-9070-c017a98118c0 [Follower]" leader= testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [WARN] server-b6323d63-d110-2b16-9070-c017a98118c0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [WARN] server-b6323d63-d110-2b16-9070-c017a98118c0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: entering candidate state: node="Node at b6323d63-d110-2b16-9070-c017a98118c0 [Candidate]" term=2 testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: entering candidate state: node="Node at b6323d63-d110-2b16-9070-c017a98118c0 [Candidate]" term=2 testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: election won: tally=2 testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: election won: tally=2 testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: entering leader state: leader="Node at b6323d63-d110-2b16-9070-c017a98118c0 [Leader]" testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: entering leader state: leader="Node at b6323d63-d110-2b16-9070-c017a98118c0 [Leader]" testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: added peer, starting replication: peer=server-a0d11a06-75af-130f-79f9-fdb8a55698cb testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: added peer, starting replication: peer=server-a0d11a06-75af-130f-79f9-fdb8a55698cb testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: added peer, starting replication: peer=server-60f8640d-a16a-48bc-a470-158993ffe7cc testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: added peer, starting replication: peer=server-60f8640d-a16a-48bc-a470-158993ffe7cc testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: pipelining replication: peer="{Voter server-60f8640d-a16a-48bc-a470-158993ffe7cc 60f8640d-a16a-48bc-a470-158993ffe7cc}" testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: pipelining replication: peer="{Voter server-60f8640d-a16a-48bc-a470-158993ffe7cc 60f8640d-a16a-48bc-a470-158993ffe7cc}" testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: pipelining replication: peer="{Voter server-a0d11a06-75af-130f-79f9-fdb8a55698cb a0d11a06-75af-130f-79f9-fdb8a55698cb}" testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.381Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: pipelining replication: peer="{Voter server-a0d11a06-75af-130f-79f9-fdb8a55698cb a0d11a06-75af-130f-79f9-fdb8a55698cb}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:19.486Z [INFO] stable state for Leader reached at 2025-02-09 16:52:19.381498821 +0000 UTC m=+13.711731509 (1 nodes), 63.481209ms from start of poll, 63.905096ms from cluster start. Timeout at 2025-02-09 16:52:19.486552596 +0000 UTC m=+13.816785282, 105.053773ms after stability testing.go:149: cluster: 2025-02-09T16:52:19.486Z [INFO] stable state for Leader reached at 2025-02-09 16:52:19.381498821 +0000 UTC m=+13.711731509 (1 nodes), 63.481209ms from start of poll, 63.905096ms from cluster start. Timeout at 2025-02-09 16:52:19.486552596 +0000 UTC m=+13.816785282, 105.053773ms after stability testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.500Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: aborting pipeline replication: peer="{Voter server-60f8640d-a16a-48bc-a470-158993ffe7cc 60f8640d-a16a-48bc-a470-158993ffe7cc}" testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.500Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: aborting pipeline replication: peer="{Voter server-60f8640d-a16a-48bc-a470-158993ffe7cc 60f8640d-a16a-48bc-a470-158993ffe7cc}" testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.500Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: aborting pipeline replication: peer="{Voter server-a0d11a06-75af-130f-79f9-fdb8a55698cb a0d11a06-75af-130f-79f9-fdb8a55698cb}" testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.500Z [INFO] server-b6323d63-d110-2b16-9070-c017a98118c0: aborting pipeline replication: peer="{Voter server-a0d11a06-75af-130f-79f9-fdb8a55698cb a0d11a06-75af-130f-79f9-fdb8a55698cb}" testLoggerAdapter verbose: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.551Z [ERROR] server-b6323d63-d110-2b16-9070-c017a98118c0: failed to heartbeat to: peer=60f8640d-a16a-48bc-a470-158993ffe7cc error="command timed out" testing.go:149: server-b6323d63-d110-2b16-9070-c017a98118c0: 2025-02-09T16:52:19.551Z [ERROR] server-b6323d63-d110-2b16-9070-c017a98118c0: failed to heartbeat to: peer=60f8640d-a16a-48bc-a470-158993ffe7cc error="command timed out" --- PASS: TestRaft_Barrier (0.23s) === RUN TestRaft_VerifyLeader testLoggerAdapter verbose: server-08b28ba4-8503-a431-9a28-87dfa6f32568: 2025-02-09T16:52:19.551Z [INFO] server-08b28ba4-8503-a431-9a28-87dfa6f32568: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08b28ba4-8503-a431-9a28-87dfa6f32568 Address:08b28ba4-8503-a431-9a28-87dfa6f32568} {Suffrage:Voter ID:server-85464139-332e-d54f-7554-0233c2e9a21d Address:85464139-332e-d54f-7554-0233c2e9a21d} {Suffrage:Voter ID:server-44912792-8f2b-4d12-c2c2-348847e75f63 Address:44912792-8f2b-4d12-c2c2-348847e75f63}]" testing.go:149: server-08b28ba4-8503-a431-9a28-87dfa6f32568: 2025-02-09T16:52:19.551Z [INFO] server-08b28ba4-8503-a431-9a28-87dfa6f32568: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08b28ba4-8503-a431-9a28-87dfa6f32568 Address:08b28ba4-8503-a431-9a28-87dfa6f32568} {Suffrage:Voter ID:server-85464139-332e-d54f-7554-0233c2e9a21d Address:85464139-332e-d54f-7554-0233c2e9a21d} {Suffrage:Voter ID:server-44912792-8f2b-4d12-c2c2-348847e75f63 Address:44912792-8f2b-4d12-c2c2-348847e75f63}]" testLoggerAdapter verbose: server-85464139-332e-d54f-7554-0233c2e9a21d: 2025-02-09T16:52:19.552Z [INFO] server-85464139-332e-d54f-7554-0233c2e9a21d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08b28ba4-8503-a431-9a28-87dfa6f32568 Address:08b28ba4-8503-a431-9a28-87dfa6f32568} {Suffrage:Voter ID:server-85464139-332e-d54f-7554-0233c2e9a21d Address:85464139-332e-d54f-7554-0233c2e9a21d} {Suffrage:Voter ID:server-44912792-8f2b-4d12-c2c2-348847e75f63 Address:44912792-8f2b-4d12-c2c2-348847e75f63}]" testing.go:149: server-85464139-332e-d54f-7554-0233c2e9a21d: 2025-02-09T16:52:19.552Z [INFO] server-85464139-332e-d54f-7554-0233c2e9a21d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08b28ba4-8503-a431-9a28-87dfa6f32568 Address:08b28ba4-8503-a431-9a28-87dfa6f32568} {Suffrage:Voter ID:server-85464139-332e-d54f-7554-0233c2e9a21d Address:85464139-332e-d54f-7554-0233c2e9a21d} {Suffrage:Voter ID:server-44912792-8f2b-4d12-c2c2-348847e75f63 Address:44912792-8f2b-4d12-c2c2-348847e75f63}]" testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.552Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08b28ba4-8503-a431-9a28-87dfa6f32568 Address:08b28ba4-8503-a431-9a28-87dfa6f32568} {Suffrage:Voter ID:server-85464139-332e-d54f-7554-0233c2e9a21d Address:85464139-332e-d54f-7554-0233c2e9a21d} {Suffrage:Voter ID:server-44912792-8f2b-4d12-c2c2-348847e75f63 Address:44912792-8f2b-4d12-c2c2-348847e75f63}]" testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.552Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08b28ba4-8503-a431-9a28-87dfa6f32568 Address:08b28ba4-8503-a431-9a28-87dfa6f32568} {Suffrage:Voter ID:server-85464139-332e-d54f-7554-0233c2e9a21d Address:85464139-332e-d54f-7554-0233c2e9a21d} {Suffrage:Voter ID:server-44912792-8f2b-4d12-c2c2-348847e75f63 Address:44912792-8f2b-4d12-c2c2-348847e75f63}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:19.552Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:19.552Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-08b28ba4-8503-a431-9a28-87dfa6f32568: 2025-02-09T16:52:19.552Z [INFO] server-08b28ba4-8503-a431-9a28-87dfa6f32568: entering follower state: follower="Node at 08b28ba4-8503-a431-9a28-87dfa6f32568 [Follower]" leader= testing.go:149: server-08b28ba4-8503-a431-9a28-87dfa6f32568: 2025-02-09T16:52:19.552Z [INFO] server-08b28ba4-8503-a431-9a28-87dfa6f32568: entering follower state: follower="Node at 08b28ba4-8503-a431-9a28-87dfa6f32568 [Follower]" leader= testLoggerAdapter verbose: server-85464139-332e-d54f-7554-0233c2e9a21d: 2025-02-09T16:52:19.552Z [INFO] server-85464139-332e-d54f-7554-0233c2e9a21d: entering follower state: follower="Node at 85464139-332e-d54f-7554-0233c2e9a21d [Follower]" leader= testing.go:149: server-85464139-332e-d54f-7554-0233c2e9a21d: 2025-02-09T16:52:19.552Z [INFO] server-85464139-332e-d54f-7554-0233c2e9a21d: entering follower state: follower="Node at 85464139-332e-d54f-7554-0233c2e9a21d [Follower]" leader= testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.552Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: entering follower state: follower="Node at 44912792-8f2b-4d12-c2c2-348847e75f63 [Follower]" leader= testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.552Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: entering follower state: follower="Node at 44912792-8f2b-4d12-c2c2-348847e75f63 [Follower]" leader= testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [WARN] server-44912792-8f2b-4d12-c2c2-348847e75f63: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [WARN] server-44912792-8f2b-4d12-c2c2-348847e75f63: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: entering candidate state: node="Node at 44912792-8f2b-4d12-c2c2-348847e75f63 [Candidate]" term=2 testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: entering candidate state: node="Node at 44912792-8f2b-4d12-c2c2-348847e75f63 [Candidate]" term=2 testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: election won: tally=2 testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: election won: tally=2 testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: entering leader state: leader="Node at 44912792-8f2b-4d12-c2c2-348847e75f63 [Leader]" testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: entering leader state: leader="Node at 44912792-8f2b-4d12-c2c2-348847e75f63 [Leader]" testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: added peer, starting replication: peer=server-08b28ba4-8503-a431-9a28-87dfa6f32568 testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: added peer, starting replication: peer=server-08b28ba4-8503-a431-9a28-87dfa6f32568 testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: added peer, starting replication: peer=server-85464139-332e-d54f-7554-0233c2e9a21d testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: added peer, starting replication: peer=server-85464139-332e-d54f-7554-0233c2e9a21d testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: pipelining replication: peer="{Voter server-85464139-332e-d54f-7554-0233c2e9a21d 85464139-332e-d54f-7554-0233c2e9a21d}" testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: pipelining replication: peer="{Voter server-85464139-332e-d54f-7554-0233c2e9a21d 85464139-332e-d54f-7554-0233c2e9a21d}" testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: pipelining replication: peer="{Voter server-08b28ba4-8503-a431-9a28-87dfa6f32568 08b28ba4-8503-a431-9a28-87dfa6f32568}" testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.607Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: pipelining replication: peer="{Voter server-08b28ba4-8503-a431-9a28-87dfa6f32568 08b28ba4-8503-a431-9a28-87dfa6f32568}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:19.713Z [INFO] stable state for Leader reached at 2025-02-09 16:52:19.607797765 +0000 UTC m=+13.938030453 (1 nodes), 55.60314ms from start of poll, 55.963802ms from cluster start. Timeout at 2025-02-09 16:52:19.712979982 +0000 UTC m=+14.043212670, 105.182217ms after stability testing.go:149: cluster: 2025-02-09T16:52:19.713Z [INFO] stable state for Leader reached at 2025-02-09 16:52:19.607797765 +0000 UTC m=+13.938030453 (1 nodes), 55.60314ms from start of poll, 55.963802ms from cluster start. Timeout at 2025-02-09 16:52:19.712979982 +0000 UTC m=+14.043212670, 105.182217ms after stability testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.713Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: aborting pipeline replication: peer="{Voter server-85464139-332e-d54f-7554-0233c2e9a21d 85464139-332e-d54f-7554-0233c2e9a21d}" testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.713Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: aborting pipeline replication: peer="{Voter server-85464139-332e-d54f-7554-0233c2e9a21d 85464139-332e-d54f-7554-0233c2e9a21d}" testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.713Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: aborting pipeline replication: peer="{Voter server-08b28ba4-8503-a431-9a28-87dfa6f32568 08b28ba4-8503-a431-9a28-87dfa6f32568}" testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.713Z [INFO] server-44912792-8f2b-4d12-c2c2-348847e75f63: aborting pipeline replication: peer="{Voter server-08b28ba4-8503-a431-9a28-87dfa6f32568 08b28ba4-8503-a431-9a28-87dfa6f32568}" testLoggerAdapter verbose: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.763Z [ERROR] server-44912792-8f2b-4d12-c2c2-348847e75f63: failed to heartbeat to: peer=08b28ba4-8503-a431-9a28-87dfa6f32568 error="command timed out" testing.go:149: server-44912792-8f2b-4d12-c2c2-348847e75f63: 2025-02-09T16:52:19.763Z [ERROR] server-44912792-8f2b-4d12-c2c2-348847e75f63: failed to heartbeat to: peer=08b28ba4-8503-a431-9a28-87dfa6f32568 error="command timed out" --- PASS: TestRaft_VerifyLeader (0.21s) === RUN TestRaft_VerifyLeader_Single testLoggerAdapter verbose: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.763Z [INFO] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc Address:403cd8cf-8b68-bea7-3980-bae8bfa40dbc}]" testing.go:149: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.763Z [INFO] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc Address:403cd8cf-8b68-bea7-3980-bae8bfa40dbc}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:19.763Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:19.763Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.763Z [INFO] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: entering follower state: follower="Node at 403cd8cf-8b68-bea7-3980-bae8bfa40dbc [Follower]" leader= testing.go:149: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.763Z [INFO] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: entering follower state: follower="Node at 403cd8cf-8b68-bea7-3980-bae8bfa40dbc [Follower]" leader= testLoggerAdapter verbose: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.822Z [WARN] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.822Z [WARN] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.823Z [INFO] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: entering candidate state: node="Node at 403cd8cf-8b68-bea7-3980-bae8bfa40dbc [Candidate]" term=2 testing.go:149: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.823Z [INFO] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: entering candidate state: node="Node at 403cd8cf-8b68-bea7-3980-bae8bfa40dbc [Candidate]" term=2 testLoggerAdapter verbose: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.823Z [INFO] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: election won: tally=1 testing.go:149: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.823Z [INFO] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: election won: tally=1 testLoggerAdapter verbose: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.823Z [INFO] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: entering leader state: leader="Node at 403cd8cf-8b68-bea7-3980-bae8bfa40dbc [Leader]" testing.go:149: server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: 2025-02-09T16:52:19.823Z [INFO] server-403cd8cf-8b68-bea7-3980-bae8bfa40dbc: entering leader state: leader="Node at 403cd8cf-8b68-bea7-3980-bae8bfa40dbc [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:19.929Z [INFO] stable state for Leader reached at 2025-02-09 16:52:19.823135772 +0000 UTC m=+14.153368460 (1 nodes), 59.150395ms from start of poll, 59.322156ms from cluster start. Timeout at 2025-02-09 16:52:19.929163277 +0000 UTC m=+14.259395966, 106.027506ms after stability testing.go:149: cluster: 2025-02-09T16:52:19.929Z [INFO] stable state for Leader reached at 2025-02-09 16:52:19.823135772 +0000 UTC m=+14.153368460 (1 nodes), 59.150395ms from start of poll, 59.322156ms from cluster start. Timeout at 2025-02-09 16:52:19.929163277 +0000 UTC m=+14.259395966, 106.027506ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.17s) === RUN TestRaft_VerifyLeader_Fail testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:19.929Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 Address:a3ec91e7-1b0c-6eb2-c17c-59ac20845b41} {Suffrage:Voter ID:server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 Address:6e8a3be0-29d5-e0e2-6b8c-4898e302fd73}]" testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:19.929Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 Address:a3ec91e7-1b0c-6eb2-c17c-59ac20845b41} {Suffrage:Voter ID:server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 Address:6e8a3be0-29d5-e0e2-6b8c-4898e302fd73}]" testLoggerAdapter verbose: server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73: 2025-02-09T16:52:19.929Z [INFO] server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 Address:a3ec91e7-1b0c-6eb2-c17c-59ac20845b41} {Suffrage:Voter ID:server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 Address:6e8a3be0-29d5-e0e2-6b8c-4898e302fd73}]" testing.go:149: server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73: 2025-02-09T16:52:19.929Z [INFO] server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 Address:a3ec91e7-1b0c-6eb2-c17c-59ac20845b41} {Suffrage:Voter ID:server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 Address:6e8a3be0-29d5-e0e2-6b8c-4898e302fd73}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:19.929Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:19.929Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:19.929Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: entering follower state: follower="Node at a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 [Follower]" leader= testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:19.929Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: entering follower state: follower="Node at a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 [Follower]" leader= testLoggerAdapter verbose: server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73: 2025-02-09T16:52:19.930Z [INFO] server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73: entering follower state: follower="Node at 6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 [Follower]" leader= testing.go:149: server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73: 2025-02-09T16:52:19.930Z [INFO] server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73: entering follower state: follower="Node at 6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 [Follower]" leader= testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [WARN] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [WARN] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: entering candidate state: node="Node at a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 [Candidate]" term=2 testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: entering candidate state: node="Node at a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 [Candidate]" term=2 testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: election won: tally=2 testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: election won: tally=2 testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: entering leader state: leader="Node at a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 [Leader]" testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: entering leader state: leader="Node at a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 [Leader]" testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: added peer, starting replication: peer=server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: added peer, starting replication: peer=server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: pipelining replication: peer="{Voter server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 6e8a3be0-29d5-e0e2-6b8c-4898e302fd73}" testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.014Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: pipelining replication: peer="{Voter server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 6e8a3be0-29d5-e0e2-6b8c-4898e302fd73}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.120Z [INFO] stable state for Leader reached at 2025-02-09 16:52:20.014413435 +0000 UTC m=+14.344646123 (1 nodes), 84.457826ms from start of poll, 84.775587ms from cluster start. Timeout at 2025-02-09 16:52:20.120082914 +0000 UTC m=+14.450315602, 105.669479ms after stability testing.go:149: cluster: 2025-02-09T16:52:20.120Z [INFO] stable state for Leader reached at 2025-02-09 16:52:20.014413435 +0000 UTC m=+14.344646123 (1 nodes), 84.457826ms from start of poll, 84.775587ms from cluster start. Timeout at 2025-02-09 16:52:20.120082914 +0000 UTC m=+14.450315602, 105.669479ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.120Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:20.120Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.225Z [INFO] stable state for Follower reached at 2025-02-09 16:52:20.12019592 +0000 UTC m=+14.450428606 (1 nodes), 869ns from start of poll, 190.55807ms from cluster start. Timeout at 2025-02-09 16:52:20.225926359 +0000 UTC m=+14.556159045, 105.730439ms after stability testing.go:149: cluster: 2025-02-09T16:52:20.225Z [INFO] stable state for Follower reached at 2025-02-09 16:52:20.12019592 +0000 UTC m=+14.450428606 (1 nodes), 869ns from start of poll, 190.55807ms from cluster start. Timeout at 2025-02-09 16:52:20.225926359 +0000 UTC m=+14.556159045, 105.730439ms after stability testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.226Z [WARN] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: new leader elected, stepping down testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.226Z [WARN] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: new leader elected, stepping down testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.226Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: entering follower state: follower="Node at a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 [Follower]" leader= testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.226Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: entering follower state: follower="Node at a3ec91e7-1b0c-6eb2-c17c-59ac20845b41 [Follower]" leader= testLoggerAdapter verbose: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.226Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: aborting pipeline replication: peer="{Voter server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 6e8a3be0-29d5-e0e2-6b8c-4898e302fd73}" testing.go:149: server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: 2025-02-09T16:52:20.226Z [INFO] server-a3ec91e7-1b0c-6eb2-c17c-59ac20845b41: aborting pipeline replication: peer="{Voter server-6e8a3be0-29d5-e0e2-6b8c-4898e302fd73 6e8a3be0-29d5-e0e2-6b8c-4898e302fd73}" --- PASS: TestRaft_VerifyLeader_Fail (0.30s) === RUN TestRaft_VerifyLeader_PartialConnect testLoggerAdapter verbose: server-301f5a03-2f96-29bc-5b48-e522fb6764f6: 2025-02-09T16:52:20.226Z [INFO] server-301f5a03-2f96-29bc-5b48-e522fb6764f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-301f5a03-2f96-29bc-5b48-e522fb6764f6 Address:301f5a03-2f96-29bc-5b48-e522fb6764f6} {Suffrage:Voter ID:server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f Address:cac67c1a-90a2-a132-dd3a-85ef2b081f0f} {Suffrage:Voter ID:server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88 Address:42b54c2d-9710-63b8-b1a2-5a3a27e78e88}]" testing.go:149: server-301f5a03-2f96-29bc-5b48-e522fb6764f6: 2025-02-09T16:52:20.226Z [INFO] server-301f5a03-2f96-29bc-5b48-e522fb6764f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-301f5a03-2f96-29bc-5b48-e522fb6764f6 Address:301f5a03-2f96-29bc-5b48-e522fb6764f6} {Suffrage:Voter ID:server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f Address:cac67c1a-90a2-a132-dd3a-85ef2b081f0f} {Suffrage:Voter ID:server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88 Address:42b54c2d-9710-63b8-b1a2-5a3a27e78e88}]" testLoggerAdapter verbose: server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f: 2025-02-09T16:52:20.226Z [INFO] server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-301f5a03-2f96-29bc-5b48-e522fb6764f6 Address:301f5a03-2f96-29bc-5b48-e522fb6764f6} {Suffrage:Voter ID:server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f Address:cac67c1a-90a2-a132-dd3a-85ef2b081f0f} {Suffrage:Voter ID:server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88 Address:42b54c2d-9710-63b8-b1a2-5a3a27e78e88}]" testing.go:149: server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f: 2025-02-09T16:52:20.226Z [INFO] server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-301f5a03-2f96-29bc-5b48-e522fb6764f6 Address:301f5a03-2f96-29bc-5b48-e522fb6764f6} {Suffrage:Voter ID:server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f Address:cac67c1a-90a2-a132-dd3a-85ef2b081f0f} {Suffrage:Voter ID:server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88 Address:42b54c2d-9710-63b8-b1a2-5a3a27e78e88}]" testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.226Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-301f5a03-2f96-29bc-5b48-e522fb6764f6 Address:301f5a03-2f96-29bc-5b48-e522fb6764f6} {Suffrage:Voter ID:server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f Address:cac67c1a-90a2-a132-dd3a-85ef2b081f0f} {Suffrage:Voter ID:server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88 Address:42b54c2d-9710-63b8-b1a2-5a3a27e78e88}]" testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.226Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-301f5a03-2f96-29bc-5b48-e522fb6764f6 Address:301f5a03-2f96-29bc-5b48-e522fb6764f6} {Suffrage:Voter ID:server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f Address:cac67c1a-90a2-a132-dd3a-85ef2b081f0f} {Suffrage:Voter ID:server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88 Address:42b54c2d-9710-63b8-b1a2-5a3a27e78e88}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.226Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:20.226Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-301f5a03-2f96-29bc-5b48-e522fb6764f6: 2025-02-09T16:52:20.226Z [INFO] server-301f5a03-2f96-29bc-5b48-e522fb6764f6: entering follower state: follower="Node at 301f5a03-2f96-29bc-5b48-e522fb6764f6 [Follower]" leader= testing.go:149: server-301f5a03-2f96-29bc-5b48-e522fb6764f6: 2025-02-09T16:52:20.226Z [INFO] server-301f5a03-2f96-29bc-5b48-e522fb6764f6: entering follower state: follower="Node at 301f5a03-2f96-29bc-5b48-e522fb6764f6 [Follower]" leader= testLoggerAdapter verbose: server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f: 2025-02-09T16:52:20.227Z [INFO] server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f: entering follower state: follower="Node at cac67c1a-90a2-a132-dd3a-85ef2b081f0f [Follower]" leader= testing.go:149: server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f: 2025-02-09T16:52:20.227Z [INFO] server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f: entering follower state: follower="Node at cac67c1a-90a2-a132-dd3a-85ef2b081f0f [Follower]" leader= testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.227Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: entering follower state: follower="Node at 42b54c2d-9710-63b8-b1a2-5a3a27e78e88 [Follower]" leader= testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.227Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: entering follower state: follower="Node at 42b54c2d-9710-63b8-b1a2-5a3a27e78e88 [Follower]" leader= testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.307Z [WARN] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.307Z [WARN] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: entering candidate state: node="Node at 42b54c2d-9710-63b8-b1a2-5a3a27e78e88 [Candidate]" term=2 testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: entering candidate state: node="Node at 42b54c2d-9710-63b8-b1a2-5a3a27e78e88 [Candidate]" term=2 testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: election won: tally=2 testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: election won: tally=2 testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: entering leader state: leader="Node at 42b54c2d-9710-63b8-b1a2-5a3a27e78e88 [Leader]" testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: entering leader state: leader="Node at 42b54c2d-9710-63b8-b1a2-5a3a27e78e88 [Leader]" testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: added peer, starting replication: peer=server-301f5a03-2f96-29bc-5b48-e522fb6764f6 testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: added peer, starting replication: peer=server-301f5a03-2f96-29bc-5b48-e522fb6764f6 testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: added peer, starting replication: peer=server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: added peer, starting replication: peer=server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: pipelining replication: peer="{Voter server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f cac67c1a-90a2-a132-dd3a-85ef2b081f0f}" testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: pipelining replication: peer="{Voter server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f cac67c1a-90a2-a132-dd3a-85ef2b081f0f}" testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: pipelining replication: peer="{Voter server-301f5a03-2f96-29bc-5b48-e522fb6764f6 301f5a03-2f96-29bc-5b48-e522fb6764f6}" testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.308Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: pipelining replication: peer="{Voter server-301f5a03-2f96-29bc-5b48-e522fb6764f6 301f5a03-2f96-29bc-5b48-e522fb6764f6}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.414Z [INFO] stable state for Leader reached at 2025-02-09 16:52:20.308433 +0000 UTC m=+14.638665688 (1 nodes), 81.450556ms from start of poll, 81.802512ms from cluster start. Timeout at 2025-02-09 16:52:20.41425244 +0000 UTC m=+14.744485128, 105.81944ms after stability testing.go:149: cluster: 2025-02-09T16:52:20.414Z [INFO] stable state for Leader reached at 2025-02-09 16:52:20.308433 +0000 UTC m=+14.638665688 (1 nodes), 81.450556ms from start of poll, 81.802512ms from cluster start. Timeout at 2025-02-09 16:52:20.41425244 +0000 UTC m=+14.744485128, 105.81944ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.420Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:20.420Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.525Z [INFO] stable state for Follower reached at 2025-02-09 16:52:20.420077359 +0000 UTC m=+14.750310045 (2 nodes), 1.731µs from start of poll, 193.446869ms from cluster start. Timeout at 2025-02-09 16:52:20.52576234 +0000 UTC m=+14.855995028, 105.684983ms after stability testing.go:149: cluster: 2025-02-09T16:52:20.525Z [INFO] stable state for Follower reached at 2025-02-09 16:52:20.420077359 +0000 UTC m=+14.750310045 (2 nodes), 1.731µs from start of poll, 193.446869ms from cluster start. Timeout at 2025-02-09 16:52:20.52576234 +0000 UTC m=+14.855995028, 105.684983ms after stability raft_test.go:1647: [INFO] Disconnecting Node at 301f5a03-2f96-29bc-5b48-e522fb6764f6 [Follower] testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.525Z [ERROR] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: failed to heartbeat to: peer=301f5a03-2f96-29bc-5b48-e522fb6764f6 error="failed to connect to peer: 301f5a03-2f96-29bc-5b48-e522fb6764f6" testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.525Z [ERROR] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: failed to heartbeat to: peer=301f5a03-2f96-29bc-5b48-e522fb6764f6 error="failed to connect to peer: 301f5a03-2f96-29bc-5b48-e522fb6764f6" testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.525Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: aborting pipeline replication: peer="{Voter server-301f5a03-2f96-29bc-5b48-e522fb6764f6 301f5a03-2f96-29bc-5b48-e522fb6764f6}" testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.525Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: aborting pipeline replication: peer="{Voter server-301f5a03-2f96-29bc-5b48-e522fb6764f6 301f5a03-2f96-29bc-5b48-e522fb6764f6}" testLoggerAdapter verbose: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.526Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: aborting pipeline replication: peer="{Voter server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f cac67c1a-90a2-a132-dd3a-85ef2b081f0f}" testing.go:149: server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: 2025-02-09T16:52:20.526Z [INFO] server-42b54c2d-9710-63b8-b1a2-5a3a27e78e88: aborting pipeline replication: peer="{Voter server-cac67c1a-90a2-a132-dd3a-85ef2b081f0f cac67c1a-90a2-a132-dd3a-85ef2b081f0f}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.30s) === RUN TestRaft_StartAsLeader testLoggerAdapter verbose: server-2eca0b9c-010a-eaa8-a681-90df706295db: 2025-02-09T16:52:20.526Z [INFO] server-2eca0b9c-010a-eaa8-a681-90df706295db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eca0b9c-010a-eaa8-a681-90df706295db Address:2eca0b9c-010a-eaa8-a681-90df706295db}]" testing.go:149: server-2eca0b9c-010a-eaa8-a681-90df706295db: 2025-02-09T16:52:20.526Z [INFO] server-2eca0b9c-010a-eaa8-a681-90df706295db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eca0b9c-010a-eaa8-a681-90df706295db Address:2eca0b9c-010a-eaa8-a681-90df706295db}]" testLoggerAdapter verbose: server-2eca0b9c-010a-eaa8-a681-90df706295db: 2025-02-09T16:52:20.526Z [INFO] server-2eca0b9c-010a-eaa8-a681-90df706295db: entering leader state: leader="Node at 2eca0b9c-010a-eaa8-a681-90df706295db [Leader]" testing.go:149: server-2eca0b9c-010a-eaa8-a681-90df706295db: 2025-02-09T16:52:20.526Z [INFO] server-2eca0b9c-010a-eaa8-a681-90df706295db: entering leader state: leader="Node at 2eca0b9c-010a-eaa8-a681-90df706295db [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh testLoggerAdapter verbose: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.526Z [INFO] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e62b659d-f5c3-975c-88ab-3505dabab6e8 Address:e62b659d-f5c3-975c-88ab-3505dabab6e8}]" testing.go:149: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.526Z [INFO] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e62b659d-f5c3-975c-88ab-3505dabab6e8 Address:e62b659d-f5c3-975c-88ab-3505dabab6e8}]" testLoggerAdapter verbose: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.526Z [INFO] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: entering follower state: follower="Node at e62b659d-f5c3-975c-88ab-3505dabab6e8 [Follower]" leader= testing.go:149: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.526Z [INFO] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: entering follower state: follower="Node at e62b659d-f5c3-975c-88ab-3505dabab6e8 [Follower]" leader= testLoggerAdapter verbose: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.612Z [WARN] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.612Z [WARN] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.612Z [INFO] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: entering candidate state: node="Node at e62b659d-f5c3-975c-88ab-3505dabab6e8 [Candidate]" term=2 testing.go:149: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.612Z [INFO] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: entering candidate state: node="Node at e62b659d-f5c3-975c-88ab-3505dabab6e8 [Candidate]" term=2 testLoggerAdapter verbose: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.612Z [INFO] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: election won: tally=1 testing.go:149: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.612Z [INFO] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: election won: tally=1 testLoggerAdapter verbose: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.612Z [INFO] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: entering leader state: leader="Node at e62b659d-f5c3-975c-88ab-3505dabab6e8 [Leader]" testing.go:149: server-e62b659d-f5c3-975c-88ab-3505dabab6e8: 2025-02-09T16:52:20.612Z [INFO] server-e62b659d-f5c3-975c-88ab-3505dabab6e8: entering leader state: leader="Node at e62b659d-f5c3-975c-88ab-3505dabab6e8 [Leader]" --- PASS: TestRaft_NotifyCh (0.09s) === RUN TestRaft_Voting testLoggerAdapter verbose: server-3be68524-fddc-467f-fdec-f67e71659a69: 2025-02-09T16:52:20.613Z [INFO] server-3be68524-fddc-467f-fdec-f67e71659a69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be68524-fddc-467f-fdec-f67e71659a69 Address:3be68524-fddc-467f-fdec-f67e71659a69} {Suffrage:Voter ID:server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d Address:b96e2dd7-85a2-89fd-d2c1-626d931bb44d} {Suffrage:Voter ID:server-43e1c16f-042f-32cd-94e1-57f16143a7fb Address:43e1c16f-042f-32cd-94e1-57f16143a7fb}]" testing.go:149: server-3be68524-fddc-467f-fdec-f67e71659a69: 2025-02-09T16:52:20.613Z [INFO] server-3be68524-fddc-467f-fdec-f67e71659a69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be68524-fddc-467f-fdec-f67e71659a69 Address:3be68524-fddc-467f-fdec-f67e71659a69} {Suffrage:Voter ID:server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d Address:b96e2dd7-85a2-89fd-d2c1-626d931bb44d} {Suffrage:Voter ID:server-43e1c16f-042f-32cd-94e1-57f16143a7fb Address:43e1c16f-042f-32cd-94e1-57f16143a7fb}]" testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.613Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be68524-fddc-467f-fdec-f67e71659a69 Address:3be68524-fddc-467f-fdec-f67e71659a69} {Suffrage:Voter ID:server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d Address:b96e2dd7-85a2-89fd-d2c1-626d931bb44d} {Suffrage:Voter ID:server-43e1c16f-042f-32cd-94e1-57f16143a7fb Address:43e1c16f-042f-32cd-94e1-57f16143a7fb}]" testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.613Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be68524-fddc-467f-fdec-f67e71659a69 Address:3be68524-fddc-467f-fdec-f67e71659a69} {Suffrage:Voter ID:server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d Address:b96e2dd7-85a2-89fd-d2c1-626d931bb44d} {Suffrage:Voter ID:server-43e1c16f-042f-32cd-94e1-57f16143a7fb Address:43e1c16f-042f-32cd-94e1-57f16143a7fb}]" testLoggerAdapter verbose: server-43e1c16f-042f-32cd-94e1-57f16143a7fb: 2025-02-09T16:52:20.613Z [INFO] server-43e1c16f-042f-32cd-94e1-57f16143a7fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be68524-fddc-467f-fdec-f67e71659a69 Address:3be68524-fddc-467f-fdec-f67e71659a69} {Suffrage:Voter ID:server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d Address:b96e2dd7-85a2-89fd-d2c1-626d931bb44d} {Suffrage:Voter ID:server-43e1c16f-042f-32cd-94e1-57f16143a7fb Address:43e1c16f-042f-32cd-94e1-57f16143a7fb}]" testing.go:149: server-43e1c16f-042f-32cd-94e1-57f16143a7fb: 2025-02-09T16:52:20.613Z [INFO] server-43e1c16f-042f-32cd-94e1-57f16143a7fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3be68524-fddc-467f-fdec-f67e71659a69 Address:3be68524-fddc-467f-fdec-f67e71659a69} {Suffrage:Voter ID:server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d Address:b96e2dd7-85a2-89fd-d2c1-626d931bb44d} {Suffrage:Voter ID:server-43e1c16f-042f-32cd-94e1-57f16143a7fb Address:43e1c16f-042f-32cd-94e1-57f16143a7fb}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.613Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:20.613Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-3be68524-fddc-467f-fdec-f67e71659a69: 2025-02-09T16:52:20.613Z [INFO] server-3be68524-fddc-467f-fdec-f67e71659a69: entering follower state: follower="Node at 3be68524-fddc-467f-fdec-f67e71659a69 [Follower]" leader= testing.go:149: server-3be68524-fddc-467f-fdec-f67e71659a69: 2025-02-09T16:52:20.613Z [INFO] server-3be68524-fddc-467f-fdec-f67e71659a69: entering follower state: follower="Node at 3be68524-fddc-467f-fdec-f67e71659a69 [Follower]" leader= testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.613Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: entering follower state: follower="Node at b96e2dd7-85a2-89fd-d2c1-626d931bb44d [Follower]" leader= testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.613Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: entering follower state: follower="Node at b96e2dd7-85a2-89fd-d2c1-626d931bb44d [Follower]" leader= testLoggerAdapter verbose: server-43e1c16f-042f-32cd-94e1-57f16143a7fb: 2025-02-09T16:52:20.613Z [INFO] server-43e1c16f-042f-32cd-94e1-57f16143a7fb: entering follower state: follower="Node at 43e1c16f-042f-32cd-94e1-57f16143a7fb [Follower]" leader= testing.go:149: server-43e1c16f-042f-32cd-94e1-57f16143a7fb: 2025-02-09T16:52:20.613Z [INFO] server-43e1c16f-042f-32cd-94e1-57f16143a7fb: entering follower state: follower="Node at 43e1c16f-042f-32cd-94e1-57f16143a7fb [Follower]" leader= testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [WARN] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [WARN] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: entering candidate state: node="Node at b96e2dd7-85a2-89fd-d2c1-626d931bb44d [Candidate]" term=2 testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: entering candidate state: node="Node at b96e2dd7-85a2-89fd-d2c1-626d931bb44d [Candidate]" term=2 testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: election won: tally=2 testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: election won: tally=2 testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: entering leader state: leader="Node at b96e2dd7-85a2-89fd-d2c1-626d931bb44d [Leader]" testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: entering leader state: leader="Node at b96e2dd7-85a2-89fd-d2c1-626d931bb44d [Leader]" testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: added peer, starting replication: peer=server-3be68524-fddc-467f-fdec-f67e71659a69 testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: added peer, starting replication: peer=server-3be68524-fddc-467f-fdec-f67e71659a69 testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: added peer, starting replication: peer=server-43e1c16f-042f-32cd-94e1-57f16143a7fb testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.680Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: added peer, starting replication: peer=server-43e1c16f-042f-32cd-94e1-57f16143a7fb testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.681Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: pipelining replication: peer="{Voter server-43e1c16f-042f-32cd-94e1-57f16143a7fb 43e1c16f-042f-32cd-94e1-57f16143a7fb}" testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.681Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: pipelining replication: peer="{Voter server-43e1c16f-042f-32cd-94e1-57f16143a7fb 43e1c16f-042f-32cd-94e1-57f16143a7fb}" testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.681Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: pipelining replication: peer="{Voter server-3be68524-fddc-467f-fdec-f67e71659a69 3be68524-fddc-467f-fdec-f67e71659a69}" testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.681Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: pipelining replication: peer="{Voter server-3be68524-fddc-467f-fdec-f67e71659a69 3be68524-fddc-467f-fdec-f67e71659a69}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.786Z [INFO] stable state for Follower reached at 2025-02-09 16:52:20.681219281 +0000 UTC m=+15.011451969 (2 nodes), 67.831459ms from start of poll, 68.194374ms from cluster start. Timeout at 2025-02-09 16:52:20.786405657 +0000 UTC m=+15.116638345, 105.186376ms after stability testing.go:149: cluster: 2025-02-09T16:52:20.786Z [INFO] stable state for Follower reached at 2025-02-09 16:52:20.681219281 +0000 UTC m=+15.011451969 (2 nodes), 67.831459ms from start of poll, 68.194374ms from cluster start. Timeout at 2025-02-09 16:52:20.786405657 +0000 UTC m=+15.116638345, 105.186376ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.786Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:20.786Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.891Z [INFO] stable state for Leader reached at 2025-02-09 16:52:20.786527505 +0000 UTC m=+15.116760191 (1 nodes), 881ns from start of poll, 173.502596ms from cluster start. Timeout at 2025-02-09 16:52:20.891865807 +0000 UTC m=+15.222098495, 105.338304ms after stability testing.go:149: cluster: 2025-02-09T16:52:20.891Z [INFO] stable state for Leader reached at 2025-02-09 16:52:20.786527505 +0000 UTC m=+15.116760191 (1 nodes), 881ns from start of poll, 173.502596ms from cluster start. Timeout at 2025-02-09 16:52:20.891865807 +0000 UTC m=+15.222098495, 105.338304ms after stability testLoggerAdapter verbose: server-43e1c16f-042f-32cd-94e1-57f16143a7fb: 2025-02-09T16:52:20.891Z [WARN] server-43e1c16f-042f-32cd-94e1-57f16143a7fb: rejecting vote request since we have a leader: from=3be68524-fddc-467f-fdec-f67e71659a69 leader=b96e2dd7-85a2-89fd-d2c1-626d931bb44d testing.go:149: server-43e1c16f-042f-32cd-94e1-57f16143a7fb: 2025-02-09T16:52:20.891Z [WARN] server-43e1c16f-042f-32cd-94e1-57f16143a7fb: rejecting vote request since we have a leader: from=3be68524-fddc-467f-fdec-f67e71659a69 leader=b96e2dd7-85a2-89fd-d2c1-626d931bb44d testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.892Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: aborting pipeline replication: peer="{Voter server-43e1c16f-042f-32cd-94e1-57f16143a7fb 43e1c16f-042f-32cd-94e1-57f16143a7fb}" testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.892Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: aborting pipeline replication: peer="{Voter server-43e1c16f-042f-32cd-94e1-57f16143a7fb 43e1c16f-042f-32cd-94e1-57f16143a7fb}" testLoggerAdapter verbose: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.892Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: aborting pipeline replication: peer="{Voter server-3be68524-fddc-467f-fdec-f67e71659a69 3be68524-fddc-467f-fdec-f67e71659a69}" testing.go:149: server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: 2025-02-09T16:52:20.892Z [INFO] server-b96e2dd7-85a2-89fd-d2c1-626d931bb44d: aborting pipeline replication: peer="{Voter server-3be68524-fddc-467f-fdec-f67e71659a69 3be68524-fddc-467f-fdec-f67e71659a69}" --- PASS: TestRaft_Voting (0.28s) === RUN TestRaft_ProtocolVersion_RejectRPC testLoggerAdapter verbose: server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad: 2025-02-09T16:52:20.892Z [INFO] server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad Address:a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad} {Suffrage:Voter ID:server-74c08026-5219-806f-4c3e-149fcd4e57b9 Address:74c08026-5219-806f-4c3e-149fcd4e57b9} {Suffrage:Voter ID:server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb Address:682dc354-60da-e5d9-6e8a-d5a7aeba1fbb}]" testing.go:149: server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad: 2025-02-09T16:52:20.892Z [INFO] server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad Address:a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad} {Suffrage:Voter ID:server-74c08026-5219-806f-4c3e-149fcd4e57b9 Address:74c08026-5219-806f-4c3e-149fcd4e57b9} {Suffrage:Voter ID:server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb Address:682dc354-60da-e5d9-6e8a-d5a7aeba1fbb}]" testLoggerAdapter verbose: server-74c08026-5219-806f-4c3e-149fcd4e57b9: 2025-02-09T16:52:20.892Z [INFO] server-74c08026-5219-806f-4c3e-149fcd4e57b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad Address:a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad} {Suffrage:Voter ID:server-74c08026-5219-806f-4c3e-149fcd4e57b9 Address:74c08026-5219-806f-4c3e-149fcd4e57b9} {Suffrage:Voter ID:server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb Address:682dc354-60da-e5d9-6e8a-d5a7aeba1fbb}]" testing.go:149: server-74c08026-5219-806f-4c3e-149fcd4e57b9: 2025-02-09T16:52:20.892Z [INFO] server-74c08026-5219-806f-4c3e-149fcd4e57b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad Address:a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad} {Suffrage:Voter ID:server-74c08026-5219-806f-4c3e-149fcd4e57b9 Address:74c08026-5219-806f-4c3e-149fcd4e57b9} {Suffrage:Voter ID:server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb Address:682dc354-60da-e5d9-6e8a-d5a7aeba1fbb}]" testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.892Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad Address:a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad} {Suffrage:Voter ID:server-74c08026-5219-806f-4c3e-149fcd4e57b9 Address:74c08026-5219-806f-4c3e-149fcd4e57b9} {Suffrage:Voter ID:server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb Address:682dc354-60da-e5d9-6e8a-d5a7aeba1fbb}]" testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.892Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad Address:a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad} {Suffrage:Voter ID:server-74c08026-5219-806f-4c3e-149fcd4e57b9 Address:74c08026-5219-806f-4c3e-149fcd4e57b9} {Suffrage:Voter ID:server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb Address:682dc354-60da-e5d9-6e8a-d5a7aeba1fbb}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:20.892Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:20.892Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad: 2025-02-09T16:52:20.892Z [INFO] server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad: entering follower state: follower="Node at a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad [Follower]" leader= testing.go:149: server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad: 2025-02-09T16:52:20.892Z [INFO] server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad: entering follower state: follower="Node at a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad [Follower]" leader= testLoggerAdapter verbose: server-74c08026-5219-806f-4c3e-149fcd4e57b9: 2025-02-09T16:52:20.893Z [INFO] server-74c08026-5219-806f-4c3e-149fcd4e57b9: entering follower state: follower="Node at 74c08026-5219-806f-4c3e-149fcd4e57b9 [Follower]" leader= testing.go:149: server-74c08026-5219-806f-4c3e-149fcd4e57b9: 2025-02-09T16:52:20.893Z [INFO] server-74c08026-5219-806f-4c3e-149fcd4e57b9: entering follower state: follower="Node at 74c08026-5219-806f-4c3e-149fcd4e57b9 [Follower]" leader= testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.893Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: entering follower state: follower="Node at 682dc354-60da-e5d9-6e8a-d5a7aeba1fbb [Follower]" leader= testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.893Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: entering follower state: follower="Node at 682dc354-60da-e5d9-6e8a-d5a7aeba1fbb [Follower]" leader= testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [WARN] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [WARN] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: entering candidate state: node="Node at 682dc354-60da-e5d9-6e8a-d5a7aeba1fbb [Candidate]" term=2 testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: entering candidate state: node="Node at 682dc354-60da-e5d9-6e8a-d5a7aeba1fbb [Candidate]" term=2 testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: election won: tally=2 testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: election won: tally=2 testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: entering leader state: leader="Node at 682dc354-60da-e5d9-6e8a-d5a7aeba1fbb [Leader]" testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: entering leader state: leader="Node at 682dc354-60da-e5d9-6e8a-d5a7aeba1fbb [Leader]" testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: added peer, starting replication: peer=server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: added peer, starting replication: peer=server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: added peer, starting replication: peer=server-74c08026-5219-806f-4c3e-149fcd4e57b9 testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: added peer, starting replication: peer=server-74c08026-5219-806f-4c3e-149fcd4e57b9 testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: pipelining replication: peer="{Voter server-74c08026-5219-806f-4c3e-149fcd4e57b9 74c08026-5219-806f-4c3e-149fcd4e57b9}" testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: pipelining replication: peer="{Voter server-74c08026-5219-806f-4c3e-149fcd4e57b9 74c08026-5219-806f-4c3e-149fcd4e57b9}" testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: pipelining replication: peer="{Voter server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad}" testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:20.964Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: pipelining replication: peer="{Voter server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.070Z [INFO] stable state for Follower reached at 2025-02-09 16:52:20.964802621 +0000 UTC m=+15.295035309 (2 nodes), 71.819117ms from start of poll, 72.170302ms from cluster start. Timeout at 2025-02-09 16:52:21.070209449 +0000 UTC m=+15.400442137, 105.406828ms after stability testing.go:149: cluster: 2025-02-09T16:52:21.070Z [INFO] stable state for Follower reached at 2025-02-09 16:52:20.964802621 +0000 UTC m=+15.295035309 (2 nodes), 71.819117ms from start of poll, 72.170302ms from cluster start. Timeout at 2025-02-09 16:52:21.070209449 +0000 UTC m=+15.400442137, 105.406828ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.070Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:21.070Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.175Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.070325701 +0000 UTC m=+15.400558387 (1 nodes), 886ns from start of poll, 177.69338ms from cluster start. Timeout at 2025-02-09 16:52:21.175768196 +0000 UTC m=+15.506000882, 105.442495ms after stability testing.go:149: cluster: 2025-02-09T16:52:21.175Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.070325701 +0000 UTC m=+15.400558387 (1 nodes), 886ns from start of poll, 177.69338ms from cluster start. Timeout at 2025-02-09 16:52:21.175768196 +0000 UTC m=+15.506000882, 105.442495ms after stability testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:21.175Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: aborting pipeline replication: peer="{Voter server-74c08026-5219-806f-4c3e-149fcd4e57b9 74c08026-5219-806f-4c3e-149fcd4e57b9}" testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:21.175Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: aborting pipeline replication: peer="{Voter server-74c08026-5219-806f-4c3e-149fcd4e57b9 74c08026-5219-806f-4c3e-149fcd4e57b9}" testLoggerAdapter verbose: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:21.175Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: aborting pipeline replication: peer="{Voter server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad}" testing.go:149: server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: 2025-02-09T16:52:21.175Z [INFO] server-682dc354-60da-e5d9-6e8a-d5a7aeba1fbb: aborting pipeline replication: peer="{Voter server-a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad a6a7d5ff-0239-4cd6-08fa-b27103f1f0ad}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.28s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.176Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7bdc695-601c-95ba-9197-d6189509ca62 Address:c7bdc695-601c-95ba-9197-d6189509ca62} {Suffrage:Voter ID:5d88de32-d2e8-e310-87e7-265905441d90 Address:5d88de32-d2e8-e310-87e7-265905441d90}]" testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.176Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7bdc695-601c-95ba-9197-d6189509ca62 Address:c7bdc695-601c-95ba-9197-d6189509ca62} {Suffrage:Voter ID:5d88de32-d2e8-e310-87e7-265905441d90 Address:5d88de32-d2e8-e310-87e7-265905441d90}]" testLoggerAdapter verbose: 5d88de32-d2e8-e310-87e7-265905441d90: 2025-02-09T16:52:21.176Z [INFO] 5d88de32-d2e8-e310-87e7-265905441d90: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7bdc695-601c-95ba-9197-d6189509ca62 Address:c7bdc695-601c-95ba-9197-d6189509ca62} {Suffrage:Voter ID:5d88de32-d2e8-e310-87e7-265905441d90 Address:5d88de32-d2e8-e310-87e7-265905441d90}]" testing.go:149: 5d88de32-d2e8-e310-87e7-265905441d90: 2025-02-09T16:52:21.176Z [INFO] 5d88de32-d2e8-e310-87e7-265905441d90: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7bdc695-601c-95ba-9197-d6189509ca62 Address:c7bdc695-601c-95ba-9197-d6189509ca62} {Suffrage:Voter ID:5d88de32-d2e8-e310-87e7-265905441d90 Address:5d88de32-d2e8-e310-87e7-265905441d90}]" testLoggerAdapter verbose: 7f7cbfad-545e-d2f6-2461-dab52d3c9988: 2025-02-09T16:52:21.176Z [INFO] 7f7cbfad-545e-d2f6-2461-dab52d3c9988: initial configuration: index=0 servers=[] testing.go:149: 7f7cbfad-545e-d2f6-2461-dab52d3c9988: 2025-02-09T16:52:21.176Z [INFO] 7f7cbfad-545e-d2f6-2461-dab52d3c9988: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.176Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:21.176Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.176Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: entering follower state: follower="Node at c7bdc695-601c-95ba-9197-d6189509ca62 [Follower]" leader= testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.176Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: entering follower state: follower="Node at c7bdc695-601c-95ba-9197-d6189509ca62 [Follower]" leader= testLoggerAdapter verbose: 5d88de32-d2e8-e310-87e7-265905441d90: 2025-02-09T16:52:21.176Z [INFO] 5d88de32-d2e8-e310-87e7-265905441d90: entering follower state: follower="Node at 5d88de32-d2e8-e310-87e7-265905441d90 [Follower]" leader= testing.go:149: 5d88de32-d2e8-e310-87e7-265905441d90: 2025-02-09T16:52:21.176Z [INFO] 5d88de32-d2e8-e310-87e7-265905441d90: entering follower state: follower="Node at 5d88de32-d2e8-e310-87e7-265905441d90 [Follower]" leader= testLoggerAdapter verbose: 7f7cbfad-545e-d2f6-2461-dab52d3c9988: 2025-02-09T16:52:21.176Z [INFO] 7f7cbfad-545e-d2f6-2461-dab52d3c9988: entering follower state: follower="Node at 7f7cbfad-545e-d2f6-2461-dab52d3c9988 [Follower]" leader= testing.go:149: 7f7cbfad-545e-d2f6-2461-dab52d3c9988: 2025-02-09T16:52:21.176Z [INFO] 7f7cbfad-545e-d2f6-2461-dab52d3c9988: entering follower state: follower="Node at 7f7cbfad-545e-d2f6-2461-dab52d3c9988 [Follower]" leader= testLoggerAdapter verbose: 7f7cbfad-545e-d2f6-2461-dab52d3c9988: 2025-02-09T16:52:21.255Z [WARN] 7f7cbfad-545e-d2f6-2461-dab52d3c9988: no known peers, aborting election testing.go:149: 7f7cbfad-545e-d2f6-2461-dab52d3c9988: 2025-02-09T16:52:21.255Z [WARN] 7f7cbfad-545e-d2f6-2461-dab52d3c9988: no known peers, aborting election testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [WARN] c7bdc695-601c-95ba-9197-d6189509ca62: heartbeat timeout reached, starting election: last-leader= testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [WARN] c7bdc695-601c-95ba-9197-d6189509ca62: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: entering candidate state: node="Node at c7bdc695-601c-95ba-9197-d6189509ca62 [Candidate]" term=2 testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: entering candidate state: node="Node at c7bdc695-601c-95ba-9197-d6189509ca62 [Candidate]" term=2 testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: election won: tally=2 testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: election won: tally=2 testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: entering leader state: leader="Node at c7bdc695-601c-95ba-9197-d6189509ca62 [Leader]" testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: entering leader state: leader="Node at c7bdc695-601c-95ba-9197-d6189509ca62 [Leader]" testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: added peer, starting replication: peer=5d88de32-d2e8-e310-87e7-265905441d90 testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: added peer, starting replication: peer=5d88de32-d2e8-e310-87e7-265905441d90 testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: pipelining replication: peer="{Voter 5d88de32-d2e8-e310-87e7-265905441d90 5d88de32-d2e8-e310-87e7-265905441d90}" testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.261Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: pipelining replication: peer="{Voter 5d88de32-d2e8-e310-87e7-265905441d90 5d88de32-d2e8-e310-87e7-265905441d90}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.366Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.261505847 +0000 UTC m=+15.591738535 (1 nodes), 84.715585ms from start of poll, 85.062224ms from cluster start. Timeout at 2025-02-09 16:52:21.366708262 +0000 UTC m=+15.696940950, 105.202415ms after stability testing.go:149: cluster: 2025-02-09T16:52:21.366Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.261505847 +0000 UTC m=+15.591738535 (1 nodes), 84.715585ms from start of poll, 85.062224ms from cluster start. Timeout at 2025-02-09 16:52:21.366708262 +0000 UTC m=+15.696940950, 105.202415ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.366Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:21.366Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.472Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.366839807 +0000 UTC m=+15.697072493 (1 nodes), 1.115µs from start of poll, 190.396182ms from cluster start. Timeout at 2025-02-09 16:52:21.472255637 +0000 UTC m=+15.802488325, 105.415832ms after stability testing.go:149: cluster: 2025-02-09T16:52:21.472Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.366839807 +0000 UTC m=+15.697072493 (1 nodes), 1.115µs from start of poll, 190.396182ms from cluster start. Timeout at 2025-02-09 16:52:21.472255637 +0000 UTC m=+15.802488325, 105.415832ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.472Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:21.472Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.577Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.472371037 +0000 UTC m=+15.802603723 (1 nodes), 696ns from start of poll, 295.927412ms from cluster start. Timeout at 2025-02-09 16:52:21.577864808 +0000 UTC m=+15.908097495, 105.493772ms after stability testing.go:149: cluster: 2025-02-09T16:52:21.577Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.472371037 +0000 UTC m=+15.802603723 (1 nodes), 696ns from start of poll, 295.927412ms from cluster start. Timeout at 2025-02-09 16:52:21.577864808 +0000 UTC m=+15.908097495, 105.493772ms after stability testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.577Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: updating configuration: command=AddStaging server-id=7f7cbfad-545e-d2f6-2461-dab52d3c9988 server-addr=7f7cbfad-545e-d2f6-2461-dab52d3c9988 servers="[{Suffrage:Voter ID:c7bdc695-601c-95ba-9197-d6189509ca62 Address:c7bdc695-601c-95ba-9197-d6189509ca62} {Suffrage:Voter ID:5d88de32-d2e8-e310-87e7-265905441d90 Address:5d88de32-d2e8-e310-87e7-265905441d90} {Suffrage:Voter ID:7f7cbfad-545e-d2f6-2461-dab52d3c9988 Address:7f7cbfad-545e-d2f6-2461-dab52d3c9988}]" testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.577Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: updating configuration: command=AddStaging server-id=7f7cbfad-545e-d2f6-2461-dab52d3c9988 server-addr=7f7cbfad-545e-d2f6-2461-dab52d3c9988 servers="[{Suffrage:Voter ID:c7bdc695-601c-95ba-9197-d6189509ca62 Address:c7bdc695-601c-95ba-9197-d6189509ca62} {Suffrage:Voter ID:5d88de32-d2e8-e310-87e7-265905441d90 Address:5d88de32-d2e8-e310-87e7-265905441d90} {Suffrage:Voter ID:7f7cbfad-545e-d2f6-2461-dab52d3c9988 Address:7f7cbfad-545e-d2f6-2461-dab52d3c9988}]" testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.578Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: added peer, starting replication: peer=7f7cbfad-545e-d2f6-2461-dab52d3c9988 testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.578Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: added peer, starting replication: peer=7f7cbfad-545e-d2f6-2461-dab52d3c9988 testLoggerAdapter verbose: 7f7cbfad-545e-d2f6-2461-dab52d3c9988: 2025-02-09T16:52:21.578Z [WARN] 7f7cbfad-545e-d2f6-2461-dab52d3c9988: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: 7f7cbfad-545e-d2f6-2461-dab52d3c9988: 2025-02-09T16:52:21.578Z [WARN] 7f7cbfad-545e-d2f6-2461-dab52d3c9988: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.578Z [WARN] c7bdc695-601c-95ba-9197-d6189509ca62: appendEntries rejected, sending older logs: peer="{Voter 7f7cbfad-545e-d2f6-2461-dab52d3c9988 7f7cbfad-545e-d2f6-2461-dab52d3c9988}" next=1 testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.578Z [WARN] c7bdc695-601c-95ba-9197-d6189509ca62: appendEntries rejected, sending older logs: peer="{Voter 7f7cbfad-545e-d2f6-2461-dab52d3c9988 7f7cbfad-545e-d2f6-2461-dab52d3c9988}" next=1 testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.578Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: pipelining replication: peer="{Voter 7f7cbfad-545e-d2f6-2461-dab52d3c9988 7f7cbfad-545e-d2f6-2461-dab52d3c9988}" testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.578Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: pipelining replication: peer="{Voter 7f7cbfad-545e-d2f6-2461-dab52d3c9988 7f7cbfad-545e-d2f6-2461-dab52d3c9988}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.578Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:21.578Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.683Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.578330442 +0000 UTC m=+15.908563128 (1 nodes), 488ns from start of poll, 401.886817ms from cluster start. Timeout at 2025-02-09 16:52:21.683412841 +0000 UTC m=+16.013645527, 105.082399ms after stability testing.go:149: cluster: 2025-02-09T16:52:21.683Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.578330442 +0000 UTC m=+15.908563128 (1 nodes), 488ns from start of poll, 401.886817ms from cluster start. Timeout at 2025-02-09 16:52:21.683412841 +0000 UTC m=+16.013645527, 105.082399ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.683Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:21.683Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.789Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.683538674 +0000 UTC m=+16.013771361 (1 nodes), 606ns from start of poll, 507.09505ms from cluster start. Timeout at 2025-02-09 16:52:21.78906047 +0000 UTC m=+16.119293159, 105.521798ms after stability testing.go:149: cluster: 2025-02-09T16:52:21.789Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.683538674 +0000 UTC m=+16.013771361 (1 nodes), 606ns from start of poll, 507.09505ms from cluster start. Timeout at 2025-02-09 16:52:21.78906047 +0000 UTC m=+16.119293159, 105.521798ms after stability testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.789Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: updating configuration: command=RemoveServer server-id=7f7cbfad-545e-d2f6-2461-dab52d3c9988 server-addr= servers="[{Suffrage:Voter ID:c7bdc695-601c-95ba-9197-d6189509ca62 Address:c7bdc695-601c-95ba-9197-d6189509ca62} {Suffrage:Voter ID:5d88de32-d2e8-e310-87e7-265905441d90 Address:5d88de32-d2e8-e310-87e7-265905441d90}]" testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.789Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: updating configuration: command=RemoveServer server-id=7f7cbfad-545e-d2f6-2461-dab52d3c9988 server-addr= servers="[{Suffrage:Voter ID:c7bdc695-601c-95ba-9197-d6189509ca62 Address:c7bdc695-601c-95ba-9197-d6189509ca62} {Suffrage:Voter ID:5d88de32-d2e8-e310-87e7-265905441d90 Address:5d88de32-d2e8-e310-87e7-265905441d90}]" testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.789Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: removed peer, stopping replication: peer=7f7cbfad-545e-d2f6-2461-dab52d3c9988 last-index=4 testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.789Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: removed peer, stopping replication: peer=7f7cbfad-545e-d2f6-2461-dab52d3c9988 last-index=4 testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.789Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: aborting pipeline replication: peer="{Voter 7f7cbfad-545e-d2f6-2461-dab52d3c9988 7f7cbfad-545e-d2f6-2461-dab52d3c9988}" testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.789Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: aborting pipeline replication: peer="{Voter 7f7cbfad-545e-d2f6-2461-dab52d3c9988 7f7cbfad-545e-d2f6-2461-dab52d3c9988}" testLoggerAdapter verbose: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.789Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: aborting pipeline replication: peer="{Voter 5d88de32-d2e8-e310-87e7-265905441d90 5d88de32-d2e8-e310-87e7-265905441d90}" testing.go:149: c7bdc695-601c-95ba-9197-d6189509ca62: 2025-02-09T16:52:21.789Z [INFO] c7bdc695-601c-95ba-9197-d6189509ca62: aborting pipeline replication: peer="{Voter 5d88de32-d2e8-e310-87e7-265905441d90 5d88de32-d2e8-e310-87e7-265905441d90}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.61s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.790Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: initial configuration: index=1 servers="[{Suffrage:Voter ID:cb71177c-0d9d-05c2-08c5-89187a942a57 Address:cb71177c-0d9d-05c2-08c5-89187a942a57} {Suffrage:Voter ID:1bf2144f-27b6-25a8-ee10-8bad08cef5b0 Address:1bf2144f-27b6-25a8-ee10-8bad08cef5b0}]" testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.790Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: initial configuration: index=1 servers="[{Suffrage:Voter ID:cb71177c-0d9d-05c2-08c5-89187a942a57 Address:cb71177c-0d9d-05c2-08c5-89187a942a57} {Suffrage:Voter ID:1bf2144f-27b6-25a8-ee10-8bad08cef5b0 Address:1bf2144f-27b6-25a8-ee10-8bad08cef5b0}]" testLoggerAdapter verbose: 1bf2144f-27b6-25a8-ee10-8bad08cef5b0: 2025-02-09T16:52:21.790Z [INFO] 1bf2144f-27b6-25a8-ee10-8bad08cef5b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:cb71177c-0d9d-05c2-08c5-89187a942a57 Address:cb71177c-0d9d-05c2-08c5-89187a942a57} {Suffrage:Voter ID:1bf2144f-27b6-25a8-ee10-8bad08cef5b0 Address:1bf2144f-27b6-25a8-ee10-8bad08cef5b0}]" testing.go:149: 1bf2144f-27b6-25a8-ee10-8bad08cef5b0: 2025-02-09T16:52:21.790Z [INFO] 1bf2144f-27b6-25a8-ee10-8bad08cef5b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:cb71177c-0d9d-05c2-08c5-89187a942a57 Address:cb71177c-0d9d-05c2-08c5-89187a942a57} {Suffrage:Voter ID:1bf2144f-27b6-25a8-ee10-8bad08cef5b0 Address:1bf2144f-27b6-25a8-ee10-8bad08cef5b0}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.790Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:21.790Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.790Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: entering follower state: follower="Node at cb71177c-0d9d-05c2-08c5-89187a942a57 [Follower]" leader= testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.790Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: entering follower state: follower="Node at cb71177c-0d9d-05c2-08c5-89187a942a57 [Follower]" leader= testLoggerAdapter verbose: 1bf2144f-27b6-25a8-ee10-8bad08cef5b0: 2025-02-09T16:52:21.790Z [INFO] 1bf2144f-27b6-25a8-ee10-8bad08cef5b0: entering follower state: follower="Node at 1bf2144f-27b6-25a8-ee10-8bad08cef5b0 [Follower]" leader= testing.go:149: 1bf2144f-27b6-25a8-ee10-8bad08cef5b0: 2025-02-09T16:52:21.790Z [INFO] 1bf2144f-27b6-25a8-ee10-8bad08cef5b0: entering follower state: follower="Node at 1bf2144f-27b6-25a8-ee10-8bad08cef5b0 [Follower]" leader= testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [WARN] cb71177c-0d9d-05c2-08c5-89187a942a57: heartbeat timeout reached, starting election: last-leader= testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [WARN] cb71177c-0d9d-05c2-08c5-89187a942a57: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: entering candidate state: node="Node at cb71177c-0d9d-05c2-08c5-89187a942a57 [Candidate]" term=2 testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: entering candidate state: node="Node at cb71177c-0d9d-05c2-08c5-89187a942a57 [Candidate]" term=2 testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: election won: tally=2 testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: election won: tally=2 testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: entering leader state: leader="Node at cb71177c-0d9d-05c2-08c5-89187a942a57 [Leader]" testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: entering leader state: leader="Node at cb71177c-0d9d-05c2-08c5-89187a942a57 [Leader]" testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: added peer, starting replication: peer=1bf2144f-27b6-25a8-ee10-8bad08cef5b0 testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: added peer, starting replication: peer=1bf2144f-27b6-25a8-ee10-8bad08cef5b0 testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: pipelining replication: peer="{Voter 1bf2144f-27b6-25a8-ee10-8bad08cef5b0 1bf2144f-27b6-25a8-ee10-8bad08cef5b0}" testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:21.876Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: pipelining replication: peer="{Voter 1bf2144f-27b6-25a8-ee10-8bad08cef5b0 1bf2144f-27b6-25a8-ee10-8bad08cef5b0}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.982Z [INFO] stable state for Follower reached at 2025-02-09 16:52:21.876704244 +0000 UTC m=+16.206936932 (1 nodes), 86.55947ms from start of poll, 86.772246ms from cluster start. Timeout at 2025-02-09 16:52:21.982517527 +0000 UTC m=+16.312750213, 105.813281ms after stability testing.go:149: cluster: 2025-02-09T16:52:21.982Z [INFO] stable state for Follower reached at 2025-02-09 16:52:21.876704244 +0000 UTC m=+16.206936932 (1 nodes), 86.55947ms from start of poll, 86.772246ms from cluster start. Timeout at 2025-02-09 16:52:21.982517527 +0000 UTC m=+16.312750213, 105.813281ms after stability testLoggerAdapter verbose: server-876eab3e-8c07-6e16-5504-5b31fb6774d7: 2025-02-09T16:52:21.982Z [INFO] server-876eab3e-8c07-6e16-5504-5b31fb6774d7: initial configuration: index=0 servers=[] testing.go:149: server-876eab3e-8c07-6e16-5504-5b31fb6774d7: 2025-02-09T16:52:21.982Z [INFO] server-876eab3e-8c07-6e16-5504-5b31fb6774d7: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2025-02-09T16:52:21.982Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:21.982Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-876eab3e-8c07-6e16-5504-5b31fb6774d7: 2025-02-09T16:52:21.982Z [INFO] server-876eab3e-8c07-6e16-5504-5b31fb6774d7: entering follower state: follower="Node at 876eab3e-8c07-6e16-5504-5b31fb6774d7 [Follower]" leader= testing.go:149: server-876eab3e-8c07-6e16-5504-5b31fb6774d7: 2025-02-09T16:52:21.982Z [INFO] server-876eab3e-8c07-6e16-5504-5b31fb6774d7: entering follower state: follower="Node at 876eab3e-8c07-6e16-5504-5b31fb6774d7 [Follower]" leader= testLoggerAdapter verbose: server-876eab3e-8c07-6e16-5504-5b31fb6774d7: 2025-02-09T16:52:22.068Z [WARN] server-876eab3e-8c07-6e16-5504-5b31fb6774d7: no known peers, aborting election testing.go:149: server-876eab3e-8c07-6e16-5504-5b31fb6774d7: 2025-02-09T16:52:22.068Z [WARN] server-876eab3e-8c07-6e16-5504-5b31fb6774d7: no known peers, aborting election testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.088Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.982872851 +0000 UTC m=+16.313105538 (1 nodes), 504ns from start of poll, 192.940852ms from cluster start. Timeout at 2025-02-09 16:52:22.088616253 +0000 UTC m=+16.418848941, 105.743403ms after stability testing.go:149: cluster: 2025-02-09T16:52:22.088Z [INFO] stable state for Leader reached at 2025-02-09 16:52:21.982872851 +0000 UTC m=+16.313105538 (1 nodes), 504ns from start of poll, 192.940852ms from cluster start. Timeout at 2025-02-09 16:52:22.088616253 +0000 UTC m=+16.418848941, 105.743403ms after stability testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.088Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: updating configuration: command=AddStaging server-id=server-876eab3e-8c07-6e16-5504-5b31fb6774d7 server-addr=876eab3e-8c07-6e16-5504-5b31fb6774d7 servers="[{Suffrage:Voter ID:cb71177c-0d9d-05c2-08c5-89187a942a57 Address:cb71177c-0d9d-05c2-08c5-89187a942a57} {Suffrage:Voter ID:1bf2144f-27b6-25a8-ee10-8bad08cef5b0 Address:1bf2144f-27b6-25a8-ee10-8bad08cef5b0} {Suffrage:Voter ID:server-876eab3e-8c07-6e16-5504-5b31fb6774d7 Address:876eab3e-8c07-6e16-5504-5b31fb6774d7}]" testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.088Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: updating configuration: command=AddStaging server-id=server-876eab3e-8c07-6e16-5504-5b31fb6774d7 server-addr=876eab3e-8c07-6e16-5504-5b31fb6774d7 servers="[{Suffrage:Voter ID:cb71177c-0d9d-05c2-08c5-89187a942a57 Address:cb71177c-0d9d-05c2-08c5-89187a942a57} {Suffrage:Voter ID:1bf2144f-27b6-25a8-ee10-8bad08cef5b0 Address:1bf2144f-27b6-25a8-ee10-8bad08cef5b0} {Suffrage:Voter ID:server-876eab3e-8c07-6e16-5504-5b31fb6774d7 Address:876eab3e-8c07-6e16-5504-5b31fb6774d7}]" testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.088Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: added peer, starting replication: peer=server-876eab3e-8c07-6e16-5504-5b31fb6774d7 testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.088Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: added peer, starting replication: peer=server-876eab3e-8c07-6e16-5504-5b31fb6774d7 testLoggerAdapter verbose: server-876eab3e-8c07-6e16-5504-5b31fb6774d7: 2025-02-09T16:52:22.088Z [WARN] server-876eab3e-8c07-6e16-5504-5b31fb6774d7: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-876eab3e-8c07-6e16-5504-5b31fb6774d7: 2025-02-09T16:52:22.088Z [WARN] server-876eab3e-8c07-6e16-5504-5b31fb6774d7: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.088Z [WARN] cb71177c-0d9d-05c2-08c5-89187a942a57: appendEntries rejected, sending older logs: peer="{Voter server-876eab3e-8c07-6e16-5504-5b31fb6774d7 876eab3e-8c07-6e16-5504-5b31fb6774d7}" next=1 testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.088Z [WARN] cb71177c-0d9d-05c2-08c5-89187a942a57: appendEntries rejected, sending older logs: peer="{Voter server-876eab3e-8c07-6e16-5504-5b31fb6774d7 876eab3e-8c07-6e16-5504-5b31fb6774d7}" next=1 testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.088Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: pipelining replication: peer="{Voter server-876eab3e-8c07-6e16-5504-5b31fb6774d7 876eab3e-8c07-6e16-5504-5b31fb6774d7}" testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.088Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: pipelining replication: peer="{Voter server-876eab3e-8c07-6e16-5504-5b31fb6774d7 876eab3e-8c07-6e16-5504-5b31fb6774d7}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.089Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:22.089Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.194Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.089091286 +0000 UTC m=+16.419323972 (1 nodes), 490ns from start of poll, 299.159286ms from cluster start. Timeout at 2025-02-09 16:52:22.194761208 +0000 UTC m=+16.524993893, 105.669921ms after stability testing.go:149: cluster: 2025-02-09T16:52:22.194Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.089091286 +0000 UTC m=+16.419323972 (1 nodes), 490ns from start of poll, 299.159286ms from cluster start. Timeout at 2025-02-09 16:52:22.194761208 +0000 UTC m=+16.524993893, 105.669921ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.194Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:22.194Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.300Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.194890141 +0000 UTC m=+16.525122826 (1 nodes), 690ns from start of poll, 404.95814ms from cluster start. Timeout at 2025-02-09 16:52:22.30064535 +0000 UTC m=+16.630878038, 105.755212ms after stability testing.go:149: cluster: 2025-02-09T16:52:22.300Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.194890141 +0000 UTC m=+16.525122826 (1 nodes), 690ns from start of poll, 404.95814ms from cluster start. Timeout at 2025-02-09 16:52:22.30064535 +0000 UTC m=+16.630878038, 105.755212ms after stability testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.300Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: updating configuration: command=RemoveServer server-id=1bf2144f-27b6-25a8-ee10-8bad08cef5b0 server-addr= servers="[{Suffrage:Voter ID:cb71177c-0d9d-05c2-08c5-89187a942a57 Address:cb71177c-0d9d-05c2-08c5-89187a942a57} {Suffrage:Voter ID:server-876eab3e-8c07-6e16-5504-5b31fb6774d7 Address:876eab3e-8c07-6e16-5504-5b31fb6774d7}]" testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.300Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: updating configuration: command=RemoveServer server-id=1bf2144f-27b6-25a8-ee10-8bad08cef5b0 server-addr= servers="[{Suffrage:Voter ID:cb71177c-0d9d-05c2-08c5-89187a942a57 Address:cb71177c-0d9d-05c2-08c5-89187a942a57} {Suffrage:Voter ID:server-876eab3e-8c07-6e16-5504-5b31fb6774d7 Address:876eab3e-8c07-6e16-5504-5b31fb6774d7}]" testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.300Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: removed peer, stopping replication: peer=1bf2144f-27b6-25a8-ee10-8bad08cef5b0 last-index=4 testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.300Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: removed peer, stopping replication: peer=1bf2144f-27b6-25a8-ee10-8bad08cef5b0 last-index=4 testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.300Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: aborting pipeline replication: peer="{Voter 1bf2144f-27b6-25a8-ee10-8bad08cef5b0 1bf2144f-27b6-25a8-ee10-8bad08cef5b0}" testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.300Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: aborting pipeline replication: peer="{Voter 1bf2144f-27b6-25a8-ee10-8bad08cef5b0 1bf2144f-27b6-25a8-ee10-8bad08cef5b0}" testLoggerAdapter verbose: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.300Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: aborting pipeline replication: peer="{Voter server-876eab3e-8c07-6e16-5504-5b31fb6774d7 876eab3e-8c07-6e16-5504-5b31fb6774d7}" testing.go:149: cb71177c-0d9d-05c2-08c5-89187a942a57: 2025-02-09T16:52:22.300Z [INFO] cb71177c-0d9d-05c2-08c5-89187a942a57: aborting pipeline replication: peer="{Voter server-876eab3e-8c07-6e16-5504-5b31fb6774d7 876eab3e-8c07-6e16-5504-5b31fb6774d7}" --- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.51s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.301Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daf78732-292a-d894-97bb-bc3d80cb2891 Address:daf78732-292a-d894-97bb-bc3d80cb2891} {Suffrage:Voter ID:server-8ed6a849-50ac-04da-d74b-8e21f4edc48e Address:8ed6a849-50ac-04da-d74b-8e21f4edc48e} {Suffrage:Voter ID:server-bde78719-276c-0151-d7d9-2f2d65f22669 Address:bde78719-276c-0151-d7d9-2f2d65f22669}]" testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.301Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daf78732-292a-d894-97bb-bc3d80cb2891 Address:daf78732-292a-d894-97bb-bc3d80cb2891} {Suffrage:Voter ID:server-8ed6a849-50ac-04da-d74b-8e21f4edc48e Address:8ed6a849-50ac-04da-d74b-8e21f4edc48e} {Suffrage:Voter ID:server-bde78719-276c-0151-d7d9-2f2d65f22669 Address:bde78719-276c-0151-d7d9-2f2d65f22669}]" testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.301Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daf78732-292a-d894-97bb-bc3d80cb2891 Address:daf78732-292a-d894-97bb-bc3d80cb2891} {Suffrage:Voter ID:server-8ed6a849-50ac-04da-d74b-8e21f4edc48e Address:8ed6a849-50ac-04da-d74b-8e21f4edc48e} {Suffrage:Voter ID:server-bde78719-276c-0151-d7d9-2f2d65f22669 Address:bde78719-276c-0151-d7d9-2f2d65f22669}]" testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.301Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daf78732-292a-d894-97bb-bc3d80cb2891 Address:daf78732-292a-d894-97bb-bc3d80cb2891} {Suffrage:Voter ID:server-8ed6a849-50ac-04da-d74b-8e21f4edc48e Address:8ed6a849-50ac-04da-d74b-8e21f4edc48e} {Suffrage:Voter ID:server-bde78719-276c-0151-d7d9-2f2d65f22669 Address:bde78719-276c-0151-d7d9-2f2d65f22669}]" testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.301Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: entering follower state: follower="Node at daf78732-292a-d894-97bb-bc3d80cb2891 [Follower]" leader= testLoggerAdapter verbose: server-bde78719-276c-0151-d7d9-2f2d65f22669: 2025-02-09T16:52:22.301Z [INFO] server-bde78719-276c-0151-d7d9-2f2d65f22669: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daf78732-292a-d894-97bb-bc3d80cb2891 Address:daf78732-292a-d894-97bb-bc3d80cb2891} {Suffrage:Voter ID:server-8ed6a849-50ac-04da-d74b-8e21f4edc48e Address:8ed6a849-50ac-04da-d74b-8e21f4edc48e} {Suffrage:Voter ID:server-bde78719-276c-0151-d7d9-2f2d65f22669 Address:bde78719-276c-0151-d7d9-2f2d65f22669}]" testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.301Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: entering follower state: follower="Node at daf78732-292a-d894-97bb-bc3d80cb2891 [Follower]" leader= testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.302Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: entering follower state: follower="Node at 8ed6a849-50ac-04da-d74b-8e21f4edc48e [Follower]" leader= testing.go:149: server-bde78719-276c-0151-d7d9-2f2d65f22669: 2025-02-09T16:52:22.301Z [INFO] server-bde78719-276c-0151-d7d9-2f2d65f22669: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daf78732-292a-d894-97bb-bc3d80cb2891 Address:daf78732-292a-d894-97bb-bc3d80cb2891} {Suffrage:Voter ID:server-8ed6a849-50ac-04da-d74b-8e21f4edc48e Address:8ed6a849-50ac-04da-d74b-8e21f4edc48e} {Suffrage:Voter ID:server-bde78719-276c-0151-d7d9-2f2d65f22669 Address:bde78719-276c-0151-d7d9-2f2d65f22669}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.302Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:22.302Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.302Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: entering follower state: follower="Node at 8ed6a849-50ac-04da-d74b-8e21f4edc48e [Follower]" leader= testLoggerAdapter verbose: server-bde78719-276c-0151-d7d9-2f2d65f22669: 2025-02-09T16:52:22.302Z [INFO] server-bde78719-276c-0151-d7d9-2f2d65f22669: entering follower state: follower="Node at bde78719-276c-0151-d7d9-2f2d65f22669 [Follower]" leader= testing.go:149: server-bde78719-276c-0151-d7d9-2f2d65f22669: 2025-02-09T16:52:22.302Z [INFO] server-bde78719-276c-0151-d7d9-2f2d65f22669: entering follower state: follower="Node at bde78719-276c-0151-d7d9-2f2d65f22669 [Follower]" leader= testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [WARN] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [WARN] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: entering candidate state: node="Node at 8ed6a849-50ac-04da-d74b-8e21f4edc48e [Candidate]" term=2 testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: entering candidate state: node="Node at 8ed6a849-50ac-04da-d74b-8e21f4edc48e [Candidate]" term=2 testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: election won: tally=2 testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: election won: tally=2 testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: entering leader state: leader="Node at 8ed6a849-50ac-04da-d74b-8e21f4edc48e [Leader]" testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: entering leader state: leader="Node at 8ed6a849-50ac-04da-d74b-8e21f4edc48e [Leader]" testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: added peer, starting replication: peer=server-daf78732-292a-d894-97bb-bc3d80cb2891 testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: added peer, starting replication: peer=server-daf78732-292a-d894-97bb-bc3d80cb2891 testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: added peer, starting replication: peer=server-bde78719-276c-0151-d7d9-2f2d65f22669 testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: added peer, starting replication: peer=server-bde78719-276c-0151-d7d9-2f2d65f22669 testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: pipelining replication: peer="{Voter server-bde78719-276c-0151-d7d9-2f2d65f22669 bde78719-276c-0151-d7d9-2f2d65f22669}" testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.369Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: pipelining replication: peer="{Voter server-bde78719-276c-0151-d7d9-2f2d65f22669 bde78719-276c-0151-d7d9-2f2d65f22669}" testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.370Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: pipelining replication: peer="{Voter server-daf78732-292a-d894-97bb-bc3d80cb2891 daf78732-292a-d894-97bb-bc3d80cb2891}" testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.370Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: pipelining replication: peer="{Voter server-daf78732-292a-d894-97bb-bc3d80cb2891 daf78732-292a-d894-97bb-bc3d80cb2891}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.475Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.370133162 +0000 UTC m=+16.700365850 (1 nodes), 68.050621ms from start of poll, 68.543527ms from cluster start. Timeout at 2025-02-09 16:52:22.475258189 +0000 UTC m=+16.805490877, 105.125027ms after stability testing.go:149: cluster: 2025-02-09T16:52:22.475Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.370133162 +0000 UTC m=+16.700365850 (1 nodes), 68.050621ms from start of poll, 68.543527ms from cluster start. Timeout at 2025-02-09 16:52:22.475258189 +0000 UTC m=+16.805490877, 105.125027ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.475Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:22.475Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.581Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.475374419 +0000 UTC m=+16.805607105 (1 nodes), 808ns from start of poll, 173.784782ms from cluster start. Timeout at 2025-02-09 16:52:22.581255761 +0000 UTC m=+16.911488447, 105.881342ms after stability testing.go:149: cluster: 2025-02-09T16:52:22.581Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.475374419 +0000 UTC m=+16.805607105 (1 nodes), 808ns from start of poll, 173.784782ms from cluster start. Timeout at 2025-02-09 16:52:22.581255761 +0000 UTC m=+16.911488447, 105.881342ms after stability testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: entering candidate state: node="Node at daf78732-292a-d894-97bb-bc3d80cb2891 [Candidate]" term=3 testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: entering candidate state: node="Node at daf78732-292a-d894-97bb-bc3d80cb2891 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.581Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:22.581Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.581Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: entering follower state: follower="Node at 8ed6a849-50ac-04da-d74b-8e21f4edc48e [Follower]" leader= testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.581Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: entering follower state: follower="Node at 8ed6a849-50ac-04da-d74b-8e21f4edc48e [Follower]" leader= testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.581Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: aborting pipeline replication: peer="{Voter server-bde78719-276c-0151-d7d9-2f2d65f22669 bde78719-276c-0151-d7d9-2f2d65f22669}" testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.581Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: aborting pipeline replication: peer="{Voter server-bde78719-276c-0151-d7d9-2f2d65f22669 bde78719-276c-0151-d7d9-2f2d65f22669}" testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: election won: tally=2 testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: election won: tally=2 testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: entering leader state: leader="Node at daf78732-292a-d894-97bb-bc3d80cb2891 [Leader]" testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: entering leader state: leader="Node at daf78732-292a-d894-97bb-bc3d80cb2891 [Leader]" testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: added peer, starting replication: peer=server-8ed6a849-50ac-04da-d74b-8e21f4edc48e testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: added peer, starting replication: peer=server-8ed6a849-50ac-04da-d74b-8e21f4edc48e testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: added peer, starting replication: peer=server-bde78719-276c-0151-d7d9-2f2d65f22669 testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: added peer, starting replication: peer=server-bde78719-276c-0151-d7d9-2f2d65f22669 testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: pipelining replication: peer="{Voter server-bde78719-276c-0151-d7d9-2f2d65f22669 bde78719-276c-0151-d7d9-2f2d65f22669}" testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: pipelining replication: peer="{Voter server-bde78719-276c-0151-d7d9-2f2d65f22669 bde78719-276c-0151-d7d9-2f2d65f22669}" testLoggerAdapter verbose: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.581Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: aborting pipeline replication: peer="{Voter server-daf78732-292a-d894-97bb-bc3d80cb2891 daf78732-292a-d894-97bb-bc3d80cb2891}" testing.go:149: server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: 2025-02-09T16:52:22.581Z [INFO] server-8ed6a849-50ac-04da-d74b-8e21f4edc48e: aborting pipeline replication: peer="{Voter server-daf78732-292a-d894-97bb-bc3d80cb2891 daf78732-292a-d894-97bb-bc3d80cb2891}" testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: pipelining replication: peer="{Voter server-8ed6a849-50ac-04da-d74b-8e21f4edc48e 8ed6a849-50ac-04da-d74b-8e21f4edc48e}" testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.581Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: pipelining replication: peer="{Voter server-8ed6a849-50ac-04da-d74b-8e21f4edc48e 8ed6a849-50ac-04da-d74b-8e21f4edc48e}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.687Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.581852666 +0000 UTC m=+16.912085354 (1 nodes), 330.798µs from start of poll, 280.263031ms from cluster start. Timeout at 2025-02-09 16:52:22.687060591 +0000 UTC m=+17.017293279, 105.207925ms after stability testing.go:149: cluster: 2025-02-09T16:52:22.687Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.581852666 +0000 UTC m=+16.912085354 (1 nodes), 330.798µs from start of poll, 280.263031ms from cluster start. Timeout at 2025-02-09 16:52:22.687060591 +0000 UTC m=+17.017293279, 105.207925ms after stability testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.687Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: aborting pipeline replication: peer="{Voter server-bde78719-276c-0151-d7d9-2f2d65f22669 bde78719-276c-0151-d7d9-2f2d65f22669}" testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.687Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: aborting pipeline replication: peer="{Voter server-bde78719-276c-0151-d7d9-2f2d65f22669 bde78719-276c-0151-d7d9-2f2d65f22669}" testLoggerAdapter verbose: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.687Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: aborting pipeline replication: peer="{Voter server-8ed6a849-50ac-04da-d74b-8e21f4edc48e 8ed6a849-50ac-04da-d74b-8e21f4edc48e}" testing.go:149: server-daf78732-292a-d894-97bb-bc3d80cb2891: 2025-02-09T16:52:22.687Z [INFO] server-daf78732-292a-d894-97bb-bc3d80cb2891: aborting pipeline replication: peer="{Voter server-8ed6a849-50ac-04da-d74b-8e21f4edc48e 8ed6a849-50ac-04da-d74b-8e21f4edc48e}" --- PASS: TestRaft_LeadershipTransfer (0.39s) === RUN TestRaft_LeadershipTransferWithOneNode testLoggerAdapter verbose: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.687Z [INFO] server-0cf0ad17-b26e-3439-e008-185bf80ab085: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cf0ad17-b26e-3439-e008-185bf80ab085 Address:0cf0ad17-b26e-3439-e008-185bf80ab085}]" testing.go:149: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.687Z [INFO] server-0cf0ad17-b26e-3439-e008-185bf80ab085: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cf0ad17-b26e-3439-e008-185bf80ab085 Address:0cf0ad17-b26e-3439-e008-185bf80ab085}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.687Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:22.687Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.687Z [INFO] server-0cf0ad17-b26e-3439-e008-185bf80ab085: entering follower state: follower="Node at 0cf0ad17-b26e-3439-e008-185bf80ab085 [Follower]" leader= testing.go:149: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.687Z [INFO] server-0cf0ad17-b26e-3439-e008-185bf80ab085: entering follower state: follower="Node at 0cf0ad17-b26e-3439-e008-185bf80ab085 [Follower]" leader= testLoggerAdapter verbose: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.741Z [WARN] server-0cf0ad17-b26e-3439-e008-185bf80ab085: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.741Z [WARN] server-0cf0ad17-b26e-3439-e008-185bf80ab085: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.741Z [INFO] server-0cf0ad17-b26e-3439-e008-185bf80ab085: entering candidate state: node="Node at 0cf0ad17-b26e-3439-e008-185bf80ab085 [Candidate]" term=2 testing.go:149: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.741Z [INFO] server-0cf0ad17-b26e-3439-e008-185bf80ab085: entering candidate state: node="Node at 0cf0ad17-b26e-3439-e008-185bf80ab085 [Candidate]" term=2 testLoggerAdapter verbose: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.741Z [INFO] server-0cf0ad17-b26e-3439-e008-185bf80ab085: election won: tally=1 testing.go:149: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.741Z [INFO] server-0cf0ad17-b26e-3439-e008-185bf80ab085: election won: tally=1 testLoggerAdapter verbose: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.741Z [INFO] server-0cf0ad17-b26e-3439-e008-185bf80ab085: entering leader state: leader="Node at 0cf0ad17-b26e-3439-e008-185bf80ab085 [Leader]" testing.go:149: server-0cf0ad17-b26e-3439-e008-185bf80ab085: 2025-02-09T16:52:22.741Z [INFO] server-0cf0ad17-b26e-3439-e008-185bf80ab085: entering leader state: leader="Node at 0cf0ad17-b26e-3439-e008-185bf80ab085 [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.847Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.742049558 +0000 UTC m=+17.072282246 (1 nodes), 54.223405ms from start of poll, 54.384769ms from cluster start. Timeout at 2025-02-09 16:52:22.847520237 +0000 UTC m=+17.177752925, 105.470679ms after stability testing.go:149: cluster: 2025-02-09T16:52:22.847Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.742049558 +0000 UTC m=+17.072282246 (1 nodes), 54.223405ms from start of poll, 54.384769ms from cluster start. Timeout at 2025-02-09 16:52:22.847520237 +0000 UTC m=+17.177752925, 105.470679ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.16s) === RUN TestRaft_LeadershipTransferWithSevenNodes testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:22.848Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:22.848Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testLoggerAdapter verbose: server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0: 2025-02-09T16:52:22.848Z [INFO] server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testing.go:149: server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0: 2025-02-09T16:52:22.848Z [INFO] server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testLoggerAdapter verbose: server-0f4025be-5174-0b44-72ba-b595d980b985: 2025-02-09T16:52:22.848Z [INFO] server-0f4025be-5174-0b44-72ba-b595d980b985: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testing.go:149: server-0f4025be-5174-0b44-72ba-b595d980b985: 2025-02-09T16:52:22.848Z [INFO] server-0f4025be-5174-0b44-72ba-b595d980b985: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.848Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.848Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testLoggerAdapter verbose: server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed: 2025-02-09T16:52:22.848Z [INFO] server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testing.go:149: server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed: 2025-02-09T16:52:22.848Z [INFO] server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testLoggerAdapter verbose: server-3cc26916-5f6b-8a84-b6b3-02666a516576: 2025-02-09T16:52:22.849Z [INFO] server-3cc26916-5f6b-8a84-b6b3-02666a516576: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testing.go:149: server-3cc26916-5f6b-8a84-b6b3-02666a516576: 2025-02-09T16:52:22.849Z [INFO] server-3cc26916-5f6b-8a84-b6b3-02666a516576: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testLoggerAdapter verbose: server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce: 2025-02-09T16:52:22.849Z [INFO] server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testing.go:149: server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce: 2025-02-09T16:52:22.849Z [INFO] server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3af693-2663-4c73-1120-f0d7708dccaf Address:ff3af693-2663-4c73-1120-f0d7708dccaf} {Suffrage:Voter ID:server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 Address:13a892b3-acfd-03c1-2fb0-cfbeb01431a0} {Suffrage:Voter ID:server-0f4025be-5174-0b44-72ba-b595d980b985 Address:0f4025be-5174-0b44-72ba-b595d980b985} {Suffrage:Voter ID:server-31dd4329-dc72-b900-0dfd-3207ef2a710c Address:31dd4329-dc72-b900-0dfd-3207ef2a710c} {Suffrage:Voter ID:server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed Address:ff3bb450-10c7-6ae2-ee02-746ea70dc9ed} {Suffrage:Voter ID:server-3cc26916-5f6b-8a84-b6b3-02666a516576 Address:3cc26916-5f6b-8a84-b6b3-02666a516576} {Suffrage:Voter ID:server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce Address:f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:22.849Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:22.849Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:22.850Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: entering follower state: follower="Node at ff3af693-2663-4c73-1120-f0d7708dccaf [Follower]" leader= testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:22.850Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: entering follower state: follower="Node at ff3af693-2663-4c73-1120-f0d7708dccaf [Follower]" leader= testLoggerAdapter verbose: server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0: 2025-02-09T16:52:22.850Z [INFO] server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0: entering follower state: follower="Node at 13a892b3-acfd-03c1-2fb0-cfbeb01431a0 [Follower]" leader= testing.go:149: server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0: 2025-02-09T16:52:22.850Z [INFO] server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0: entering follower state: follower="Node at 13a892b3-acfd-03c1-2fb0-cfbeb01431a0 [Follower]" leader= testLoggerAdapter verbose: server-0f4025be-5174-0b44-72ba-b595d980b985: 2025-02-09T16:52:22.850Z [INFO] server-0f4025be-5174-0b44-72ba-b595d980b985: entering follower state: follower="Node at 0f4025be-5174-0b44-72ba-b595d980b985 [Follower]" leader= testing.go:149: server-0f4025be-5174-0b44-72ba-b595d980b985: 2025-02-09T16:52:22.850Z [INFO] server-0f4025be-5174-0b44-72ba-b595d980b985: entering follower state: follower="Node at 0f4025be-5174-0b44-72ba-b595d980b985 [Follower]" leader= testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.850Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: entering follower state: follower="Node at 31dd4329-dc72-b900-0dfd-3207ef2a710c [Follower]" leader= testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.850Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: entering follower state: follower="Node at 31dd4329-dc72-b900-0dfd-3207ef2a710c [Follower]" leader= testLoggerAdapter verbose: server-3cc26916-5f6b-8a84-b6b3-02666a516576: 2025-02-09T16:52:22.851Z [INFO] server-3cc26916-5f6b-8a84-b6b3-02666a516576: entering follower state: follower="Node at 3cc26916-5f6b-8a84-b6b3-02666a516576 [Follower]" leader= testing.go:149: server-3cc26916-5f6b-8a84-b6b3-02666a516576: 2025-02-09T16:52:22.851Z [INFO] server-3cc26916-5f6b-8a84-b6b3-02666a516576: entering follower state: follower="Node at 3cc26916-5f6b-8a84-b6b3-02666a516576 [Follower]" leader= testLoggerAdapter verbose: server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed: 2025-02-09T16:52:22.851Z [INFO] server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed: entering follower state: follower="Node at ff3bb450-10c7-6ae2-ee02-746ea70dc9ed [Follower]" leader= testing.go:149: server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed: 2025-02-09T16:52:22.851Z [INFO] server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed: entering follower state: follower="Node at ff3bb450-10c7-6ae2-ee02-746ea70dc9ed [Follower]" leader= testLoggerAdapter verbose: server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce: 2025-02-09T16:52:22.851Z [INFO] server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce: entering follower state: follower="Node at f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce [Follower]" leader= testing.go:149: server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce: 2025-02-09T16:52:22.851Z [INFO] server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce: entering follower state: follower="Node at f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce [Follower]" leader= testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.909Z [WARN] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.909Z [WARN] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: entering candidate state: node="Node at 31dd4329-dc72-b900-0dfd-3207ef2a710c [Candidate]" term=2 testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: entering candidate state: node="Node at 31dd4329-dc72-b900-0dfd-3207ef2a710c [Candidate]" term=2 testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: election won: tally=4 testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: election won: tally=4 testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: entering leader state: leader="Node at 31dd4329-dc72-b900-0dfd-3207ef2a710c [Leader]" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: entering leader state: leader="Node at 31dd4329-dc72-b900-0dfd-3207ef2a710c [Leader]" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-ff3af693-2663-4c73-1120-f0d7708dccaf testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-ff3af693-2663-4c73-1120-f0d7708dccaf testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-0f4025be-5174-0b44-72ba-b595d980b985 testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-0f4025be-5174-0b44-72ba-b595d980b985 testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-3cc26916-5f6b-8a84-b6b3-02666a516576 testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-3cc26916-5f6b-8a84-b6b3-02666a516576 testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: added peer, starting replication: peer=server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-ff3af693-2663-4c73-1120-f0d7708dccaf ff3af693-2663-4c73-1120-f0d7708dccaf}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-ff3af693-2663-4c73-1120-f0d7708dccaf ff3af693-2663-4c73-1120-f0d7708dccaf}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 13a892b3-acfd-03c1-2fb0-cfbeb01431a0}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 13a892b3-acfd-03c1-2fb0-cfbeb01431a0}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-0f4025be-5174-0b44-72ba-b595d980b985 0f4025be-5174-0b44-72ba-b595d980b985}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-0f4025be-5174-0b44-72ba-b595d980b985 0f4025be-5174-0b44-72ba-b595d980b985}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed ff3bb450-10c7-6ae2-ee02-746ea70dc9ed}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed ff3bb450-10c7-6ae2-ee02-746ea70dc9ed}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-3cc26916-5f6b-8a84-b6b3-02666a516576 3cc26916-5f6b-8a84-b6b3-02666a516576}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:22.910Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: pipelining replication: peer="{Voter server-3cc26916-5f6b-8a84-b6b3-02666a516576 3cc26916-5f6b-8a84-b6b3-02666a516576}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.016Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.910512232 +0000 UTC m=+17.240744920 (1 nodes), 60.889999ms from start of poll, 62.248601ms from cluster start. Timeout at 2025-02-09 16:52:23.016222962 +0000 UTC m=+17.346455650, 105.71073ms after stability testing.go:149: cluster: 2025-02-09T16:52:23.016Z [INFO] stable state for Leader reached at 2025-02-09 16:52:22.910512232 +0000 UTC m=+17.240744920 (1 nodes), 60.889999ms from start of poll, 62.248601ms from cluster start. Timeout at 2025-02-09 16:52:23.016222962 +0000 UTC m=+17.346455650, 105.71073ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.016Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:23.016Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.121Z [INFO] stable state for Follower reached at 2025-02-09 16:52:23.016360432 +0000 UTC m=+17.346593117 (6 nodes), 3.011µs from start of poll, 168.096798ms from cluster start. Timeout at 2025-02-09 16:52:23.121625638 +0000 UTC m=+17.451858326, 105.265209ms after stability testing.go:149: cluster: 2025-02-09T16:52:23.121Z [INFO] stable state for Follower reached at 2025-02-09 16:52:23.016360432 +0000 UTC m=+17.346593117 (6 nodes), 3.011µs from start of poll, 168.096798ms from cluster start. Timeout at 2025-02-09 16:52:23.121625638 +0000 UTC m=+17.451858326, 105.265209ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.121Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:23.121Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.227Z [INFO] stable state for Leader reached at 2025-02-09 16:52:23.121762037 +0000 UTC m=+17.451994723 (1 nodes), 1.34µs from start of poll, 273.498404ms from cluster start. Timeout at 2025-02-09 16:52:23.227196324 +0000 UTC m=+17.557429012, 105.434289ms after stability testing.go:149: cluster: 2025-02-09T16:52:23.227Z [INFO] stable state for Leader reached at 2025-02-09 16:52:23.121762037 +0000 UTC m=+17.451994723 (1 nodes), 1.34µs from start of poll, 273.498404ms from cluster start. Timeout at 2025-02-09 16:52:23.227196324 +0000 UTC m=+17.557429012, 105.434289ms after stability testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: entering candidate state: node="Node at ff3af693-2663-4c73-1120-f0d7708dccaf [Candidate]" term=3 testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: entering candidate state: node="Node at ff3af693-2663-4c73-1120-f0d7708dccaf [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.227Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:23.227Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: election won: tally=4 testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: election won: tally=4 testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: entering leader state: leader="Node at ff3af693-2663-4c73-1120-f0d7708dccaf [Leader]" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: entering leader state: leader="Node at ff3af693-2663-4c73-1120-f0d7708dccaf [Leader]" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-0f4025be-5174-0b44-72ba-b595d980b985 testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-0f4025be-5174-0b44-72ba-b595d980b985 testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-31dd4329-dc72-b900-0dfd-3207ef2a710c testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-31dd4329-dc72-b900-0dfd-3207ef2a710c testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-3cc26916-5f6b-8a84-b6b3-02666a516576 testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-3cc26916-5f6b-8a84-b6b3-02666a516576 testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: added peer, starting replication: peer=server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.227Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: entering follower state: follower="Node at 31dd4329-dc72-b900-0dfd-3207ef2a710c [Follower]" leader= testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.227Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: entering follower state: follower="Node at 31dd4329-dc72-b900-0dfd-3207ef2a710c [Follower]" leader= testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.227Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed ff3bb450-10c7-6ae2-ee02-746ea70dc9ed}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.227Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed ff3bb450-10c7-6ae2-ee02-746ea70dc9ed}" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 13a892b3-acfd-03c1-2fb0-cfbeb01431a0}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 13a892b3-acfd-03c1-2fb0-cfbeb01431a0}" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-0f4025be-5174-0b44-72ba-b595d980b985 0f4025be-5174-0b44-72ba-b595d980b985}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-0f4025be-5174-0b44-72ba-b595d980b985 0f4025be-5174-0b44-72ba-b595d980b985}" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-31dd4329-dc72-b900-0dfd-3207ef2a710c 31dd4329-dc72-b900-0dfd-3207ef2a710c}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-31dd4329-dc72-b900-0dfd-3207ef2a710c 31dd4329-dc72-b900-0dfd-3207ef2a710c}" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed ff3bb450-10c7-6ae2-ee02-746ea70dc9ed}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.227Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed ff3bb450-10c7-6ae2-ee02-746ea70dc9ed}" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.228Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-3cc26916-5f6b-8a84-b6b3-02666a516576 3cc26916-5f6b-8a84-b6b3-02666a516576}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.228Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: pipelining replication: peer="{Voter server-3cc26916-5f6b-8a84-b6b3-02666a516576 3cc26916-5f6b-8a84-b6b3-02666a516576}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.228Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-3cc26916-5f6b-8a84-b6b3-02666a516576 3cc26916-5f6b-8a84-b6b3-02666a516576}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.228Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-3cc26916-5f6b-8a84-b6b3-02666a516576 3cc26916-5f6b-8a84-b6b3-02666a516576}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.228Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.228Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.228Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-ff3af693-2663-4c73-1120-f0d7708dccaf ff3af693-2663-4c73-1120-f0d7708dccaf}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.228Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-ff3af693-2663-4c73-1120-f0d7708dccaf ff3af693-2663-4c73-1120-f0d7708dccaf}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.228Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 13a892b3-acfd-03c1-2fb0-cfbeb01431a0}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.228Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 13a892b3-acfd-03c1-2fb0-cfbeb01431a0}" testLoggerAdapter verbose: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.228Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-0f4025be-5174-0b44-72ba-b595d980b985 0f4025be-5174-0b44-72ba-b595d980b985}" testing.go:149: server-31dd4329-dc72-b900-0dfd-3207ef2a710c: 2025-02-09T16:52:23.228Z [INFO] server-31dd4329-dc72-b900-0dfd-3207ef2a710c: aborting pipeline replication: peer="{Voter server-0f4025be-5174-0b44-72ba-b595d980b985 0f4025be-5174-0b44-72ba-b595d980b985}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.333Z [INFO] stable state for Leader reached at 2025-02-09 16:52:23.227812891 +0000 UTC m=+17.558045579 (1 nodes), 393.432µs from start of poll, 379.54926ms from cluster start. Timeout at 2025-02-09 16:52:23.333380749 +0000 UTC m=+17.663613435, 105.567856ms after stability testing.go:149: cluster: 2025-02-09T16:52:23.333Z [INFO] stable state for Leader reached at 2025-02-09 16:52:23.227812891 +0000 UTC m=+17.558045579 (1 nodes), 393.432µs from start of poll, 379.54926ms from cluster start. Timeout at 2025-02-09 16:52:23.333380749 +0000 UTC m=+17.663613435, 105.567856ms after stability testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-0f4025be-5174-0b44-72ba-b595d980b985 0f4025be-5174-0b44-72ba-b595d980b985}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-0f4025be-5174-0b44-72ba-b595d980b985 0f4025be-5174-0b44-72ba-b595d980b985}" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-31dd4329-dc72-b900-0dfd-3207ef2a710c 31dd4329-dc72-b900-0dfd-3207ef2a710c}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-31dd4329-dc72-b900-0dfd-3207ef2a710c 31dd4329-dc72-b900-0dfd-3207ef2a710c}" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed ff3bb450-10c7-6ae2-ee02-746ea70dc9ed}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-ff3bb450-10c7-6ae2-ee02-746ea70dc9ed ff3bb450-10c7-6ae2-ee02-746ea70dc9ed}" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-3cc26916-5f6b-8a84-b6b3-02666a516576 3cc26916-5f6b-8a84-b6b3-02666a516576}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-3cc26916-5f6b-8a84-b6b3-02666a516576 3cc26916-5f6b-8a84-b6b3-02666a516576}" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce f5ed09c6-27fe-8bf2-7995-d7aeed6f55ce}" testLoggerAdapter verbose: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 13a892b3-acfd-03c1-2fb0-cfbeb01431a0}" testing.go:149: server-ff3af693-2663-4c73-1120-f0d7708dccaf: 2025-02-09T16:52:23.333Z [INFO] server-ff3af693-2663-4c73-1120-f0d7708dccaf: aborting pipeline replication: peer="{Voter server-13a892b3-acfd-03c1-2fb0-cfbeb01431a0 13a892b3-acfd-03c1-2fb0-cfbeb01431a0}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s) === RUN TestRaft_LeadershipTransferToInvalidID testLoggerAdapter verbose: server-36711481-d8d1-a742-bc90-9c6178f7c44a: 2025-02-09T16:52:23.334Z [INFO] server-36711481-d8d1-a742-bc90-9c6178f7c44a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36711481-d8d1-a742-bc90-9c6178f7c44a Address:36711481-d8d1-a742-bc90-9c6178f7c44a} {Suffrage:Voter ID:server-4aaa030e-ab84-021a-1acc-7101959b390f Address:4aaa030e-ab84-021a-1acc-7101959b390f} {Suffrage:Voter ID:server-452e476a-64bb-d833-1585-badf05e101de Address:452e476a-64bb-d833-1585-badf05e101de}]" testing.go:149: server-36711481-d8d1-a742-bc90-9c6178f7c44a: 2025-02-09T16:52:23.334Z [INFO] server-36711481-d8d1-a742-bc90-9c6178f7c44a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36711481-d8d1-a742-bc90-9c6178f7c44a Address:36711481-d8d1-a742-bc90-9c6178f7c44a} {Suffrage:Voter ID:server-4aaa030e-ab84-021a-1acc-7101959b390f Address:4aaa030e-ab84-021a-1acc-7101959b390f} {Suffrage:Voter ID:server-452e476a-64bb-d833-1585-badf05e101de Address:452e476a-64bb-d833-1585-badf05e101de}]" testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.334Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36711481-d8d1-a742-bc90-9c6178f7c44a Address:36711481-d8d1-a742-bc90-9c6178f7c44a} {Suffrage:Voter ID:server-4aaa030e-ab84-021a-1acc-7101959b390f Address:4aaa030e-ab84-021a-1acc-7101959b390f} {Suffrage:Voter ID:server-452e476a-64bb-d833-1585-badf05e101de Address:452e476a-64bb-d833-1585-badf05e101de}]" testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.334Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36711481-d8d1-a742-bc90-9c6178f7c44a Address:36711481-d8d1-a742-bc90-9c6178f7c44a} {Suffrage:Voter ID:server-4aaa030e-ab84-021a-1acc-7101959b390f Address:4aaa030e-ab84-021a-1acc-7101959b390f} {Suffrage:Voter ID:server-452e476a-64bb-d833-1585-badf05e101de Address:452e476a-64bb-d833-1585-badf05e101de}]" testLoggerAdapter verbose: server-452e476a-64bb-d833-1585-badf05e101de: 2025-02-09T16:52:23.334Z [INFO] server-452e476a-64bb-d833-1585-badf05e101de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36711481-d8d1-a742-bc90-9c6178f7c44a Address:36711481-d8d1-a742-bc90-9c6178f7c44a} {Suffrage:Voter ID:server-4aaa030e-ab84-021a-1acc-7101959b390f Address:4aaa030e-ab84-021a-1acc-7101959b390f} {Suffrage:Voter ID:server-452e476a-64bb-d833-1585-badf05e101de Address:452e476a-64bb-d833-1585-badf05e101de}]" testing.go:149: server-452e476a-64bb-d833-1585-badf05e101de: 2025-02-09T16:52:23.334Z [INFO] server-452e476a-64bb-d833-1585-badf05e101de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36711481-d8d1-a742-bc90-9c6178f7c44a Address:36711481-d8d1-a742-bc90-9c6178f7c44a} {Suffrage:Voter ID:server-4aaa030e-ab84-021a-1acc-7101959b390f Address:4aaa030e-ab84-021a-1acc-7101959b390f} {Suffrage:Voter ID:server-452e476a-64bb-d833-1585-badf05e101de Address:452e476a-64bb-d833-1585-badf05e101de}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.334Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:23.334Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-36711481-d8d1-a742-bc90-9c6178f7c44a: 2025-02-09T16:52:23.334Z [INFO] server-36711481-d8d1-a742-bc90-9c6178f7c44a: entering follower state: follower="Node at 36711481-d8d1-a742-bc90-9c6178f7c44a [Follower]" leader= testing.go:149: server-36711481-d8d1-a742-bc90-9c6178f7c44a: 2025-02-09T16:52:23.334Z [INFO] server-36711481-d8d1-a742-bc90-9c6178f7c44a: entering follower state: follower="Node at 36711481-d8d1-a742-bc90-9c6178f7c44a [Follower]" leader= testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.334Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: entering follower state: follower="Node at 4aaa030e-ab84-021a-1acc-7101959b390f [Follower]" leader= testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.334Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: entering follower state: follower="Node at 4aaa030e-ab84-021a-1acc-7101959b390f [Follower]" leader= testLoggerAdapter verbose: server-452e476a-64bb-d833-1585-badf05e101de: 2025-02-09T16:52:23.334Z [INFO] server-452e476a-64bb-d833-1585-badf05e101de: entering follower state: follower="Node at 452e476a-64bb-d833-1585-badf05e101de [Follower]" leader= testing.go:149: server-452e476a-64bb-d833-1585-badf05e101de: 2025-02-09T16:52:23.334Z [INFO] server-452e476a-64bb-d833-1585-badf05e101de: entering follower state: follower="Node at 452e476a-64bb-d833-1585-badf05e101de [Follower]" leader= testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.393Z [WARN] server-4aaa030e-ab84-021a-1acc-7101959b390f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.393Z [WARN] server-4aaa030e-ab84-021a-1acc-7101959b390f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.393Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: entering candidate state: node="Node at 4aaa030e-ab84-021a-1acc-7101959b390f [Candidate]" term=2 testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.393Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: entering candidate state: node="Node at 4aaa030e-ab84-021a-1acc-7101959b390f [Candidate]" term=2 testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.393Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: election won: tally=2 testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.393Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: election won: tally=2 testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.393Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: entering leader state: leader="Node at 4aaa030e-ab84-021a-1acc-7101959b390f [Leader]" testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.393Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: entering leader state: leader="Node at 4aaa030e-ab84-021a-1acc-7101959b390f [Leader]" testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.393Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: added peer, starting replication: peer=server-36711481-d8d1-a742-bc90-9c6178f7c44a testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.393Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: added peer, starting replication: peer=server-36711481-d8d1-a742-bc90-9c6178f7c44a testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.394Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: added peer, starting replication: peer=server-452e476a-64bb-d833-1585-badf05e101de testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.394Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: added peer, starting replication: peer=server-452e476a-64bb-d833-1585-badf05e101de testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.394Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: pipelining replication: peer="{Voter server-452e476a-64bb-d833-1585-badf05e101de 452e476a-64bb-d833-1585-badf05e101de}" testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.394Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: pipelining replication: peer="{Voter server-452e476a-64bb-d833-1585-badf05e101de 452e476a-64bb-d833-1585-badf05e101de}" testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.394Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: pipelining replication: peer="{Voter server-36711481-d8d1-a742-bc90-9c6178f7c44a 36711481-d8d1-a742-bc90-9c6178f7c44a}" testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.394Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: pipelining replication: peer="{Voter server-36711481-d8d1-a742-bc90-9c6178f7c44a 36711481-d8d1-a742-bc90-9c6178f7c44a}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.499Z [INFO] stable state for Leader reached at 2025-02-09 16:52:23.394226414 +0000 UTC m=+17.724459102 (1 nodes), 59.374544ms from start of poll, 59.707206ms from cluster start. Timeout at 2025-02-09 16:52:23.499229907 +0000 UTC m=+17.829462595, 105.003493ms after stability testing.go:149: cluster: 2025-02-09T16:52:23.499Z [INFO] stable state for Leader reached at 2025-02-09 16:52:23.394226414 +0000 UTC m=+17.724459102 (1 nodes), 59.374544ms from start of poll, 59.707206ms from cluster start. Timeout at 2025-02-09 16:52:23.499229907 +0000 UTC m=+17.829462595, 105.003493ms after stability testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.499Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: aborting pipeline replication: peer="{Voter server-452e476a-64bb-d833-1585-badf05e101de 452e476a-64bb-d833-1585-badf05e101de}" testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.499Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: aborting pipeline replication: peer="{Voter server-452e476a-64bb-d833-1585-badf05e101de 452e476a-64bb-d833-1585-badf05e101de}" testLoggerAdapter verbose: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.499Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: aborting pipeline replication: peer="{Voter server-36711481-d8d1-a742-bc90-9c6178f7c44a 36711481-d8d1-a742-bc90-9c6178f7c44a}" testing.go:149: server-4aaa030e-ab84-021a-1acc-7101959b390f: 2025-02-09T16:52:23.499Z [INFO] server-4aaa030e-ab84-021a-1acc-7101959b390f: aborting pipeline replication: peer="{Voter server-36711481-d8d1-a742-bc90-9c6178f7c44a 36711481-d8d1-a742-bc90-9c6178f7c44a}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s) === RUN TestRaft_LeadershipTransferToInvalidAddress testLoggerAdapter verbose: server-7b71f52c-6c43-55cd-99e7-25a5c158e819: 2025-02-09T16:52:23.500Z [INFO] server-7b71f52c-6c43-55cd-99e7-25a5c158e819: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b71f52c-6c43-55cd-99e7-25a5c158e819 Address:7b71f52c-6c43-55cd-99e7-25a5c158e819} {Suffrage:Voter ID:server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f Address:e87cd036-16ed-bbe5-c7b4-b2555fd66b3f} {Suffrage:Voter ID:server-65b2dac5-96e3-ea00-e0af-1b2edb929829 Address:65b2dac5-96e3-ea00-e0af-1b2edb929829}]" testing.go:149: server-7b71f52c-6c43-55cd-99e7-25a5c158e819: 2025-02-09T16:52:23.500Z [INFO] server-7b71f52c-6c43-55cd-99e7-25a5c158e819: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b71f52c-6c43-55cd-99e7-25a5c158e819 Address:7b71f52c-6c43-55cd-99e7-25a5c158e819} {Suffrage:Voter ID:server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f Address:e87cd036-16ed-bbe5-c7b4-b2555fd66b3f} {Suffrage:Voter ID:server-65b2dac5-96e3-ea00-e0af-1b2edb929829 Address:65b2dac5-96e3-ea00-e0af-1b2edb929829}]" testLoggerAdapter verbose: server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f: 2025-02-09T16:52:23.500Z [INFO] server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b71f52c-6c43-55cd-99e7-25a5c158e819 Address:7b71f52c-6c43-55cd-99e7-25a5c158e819} {Suffrage:Voter ID:server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f Address:e87cd036-16ed-bbe5-c7b4-b2555fd66b3f} {Suffrage:Voter ID:server-65b2dac5-96e3-ea00-e0af-1b2edb929829 Address:65b2dac5-96e3-ea00-e0af-1b2edb929829}]" testing.go:149: server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f: 2025-02-09T16:52:23.500Z [INFO] server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b71f52c-6c43-55cd-99e7-25a5c158e819 Address:7b71f52c-6c43-55cd-99e7-25a5c158e819} {Suffrage:Voter ID:server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f Address:e87cd036-16ed-bbe5-c7b4-b2555fd66b3f} {Suffrage:Voter ID:server-65b2dac5-96e3-ea00-e0af-1b2edb929829 Address:65b2dac5-96e3-ea00-e0af-1b2edb929829}]" testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.500Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b71f52c-6c43-55cd-99e7-25a5c158e819 Address:7b71f52c-6c43-55cd-99e7-25a5c158e819} {Suffrage:Voter ID:server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f Address:e87cd036-16ed-bbe5-c7b4-b2555fd66b3f} {Suffrage:Voter ID:server-65b2dac5-96e3-ea00-e0af-1b2edb929829 Address:65b2dac5-96e3-ea00-e0af-1b2edb929829}]" testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.500Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b71f52c-6c43-55cd-99e7-25a5c158e819 Address:7b71f52c-6c43-55cd-99e7-25a5c158e819} {Suffrage:Voter ID:server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f Address:e87cd036-16ed-bbe5-c7b4-b2555fd66b3f} {Suffrage:Voter ID:server-65b2dac5-96e3-ea00-e0af-1b2edb929829 Address:65b2dac5-96e3-ea00-e0af-1b2edb929829}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.500Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:23.500Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-7b71f52c-6c43-55cd-99e7-25a5c158e819: 2025-02-09T16:52:23.500Z [INFO] server-7b71f52c-6c43-55cd-99e7-25a5c158e819: entering follower state: follower="Node at 7b71f52c-6c43-55cd-99e7-25a5c158e819 [Follower]" leader= testing.go:149: server-7b71f52c-6c43-55cd-99e7-25a5c158e819: 2025-02-09T16:52:23.500Z [INFO] server-7b71f52c-6c43-55cd-99e7-25a5c158e819: entering follower state: follower="Node at 7b71f52c-6c43-55cd-99e7-25a5c158e819 [Follower]" leader= testLoggerAdapter verbose: server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f: 2025-02-09T16:52:23.500Z [INFO] server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f: entering follower state: follower="Node at e87cd036-16ed-bbe5-c7b4-b2555fd66b3f [Follower]" leader= testing.go:149: server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f: 2025-02-09T16:52:23.500Z [INFO] server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f: entering follower state: follower="Node at e87cd036-16ed-bbe5-c7b4-b2555fd66b3f [Follower]" leader= testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.500Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: entering follower state: follower="Node at 65b2dac5-96e3-ea00-e0af-1b2edb929829 [Follower]" leader= testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.500Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: entering follower state: follower="Node at 65b2dac5-96e3-ea00-e0af-1b2edb929829 [Follower]" leader= testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.554Z [WARN] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.554Z [WARN] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.554Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: entering candidate state: node="Node at 65b2dac5-96e3-ea00-e0af-1b2edb929829 [Candidate]" term=2 testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.554Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: entering candidate state: node="Node at 65b2dac5-96e3-ea00-e0af-1b2edb929829 [Candidate]" term=2 testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: election won: tally=2 testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: election won: tally=2 testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: entering leader state: leader="Node at 65b2dac5-96e3-ea00-e0af-1b2edb929829 [Leader]" testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: entering leader state: leader="Node at 65b2dac5-96e3-ea00-e0af-1b2edb929829 [Leader]" testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: added peer, starting replication: peer=server-7b71f52c-6c43-55cd-99e7-25a5c158e819 testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: added peer, starting replication: peer=server-7b71f52c-6c43-55cd-99e7-25a5c158e819 testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: added peer, starting replication: peer=server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: added peer, starting replication: peer=server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: pipelining replication: peer="{Voter server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f e87cd036-16ed-bbe5-c7b4-b2555fd66b3f}" testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: pipelining replication: peer="{Voter server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f e87cd036-16ed-bbe5-c7b4-b2555fd66b3f}" testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: pipelining replication: peer="{Voter server-7b71f52c-6c43-55cd-99e7-25a5c158e819 7b71f52c-6c43-55cd-99e7-25a5c158e819}" testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.555Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: pipelining replication: peer="{Voter server-7b71f52c-6c43-55cd-99e7-25a5c158e819 7b71f52c-6c43-55cd-99e7-25a5c158e819}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.660Z [INFO] stable state for Follower reached at 2025-02-09 16:52:23.555310022 +0000 UTC m=+17.885542710 (2 nodes), 54.980246ms from start of poll, 55.317653ms from cluster start. Timeout at 2025-02-09 16:52:23.66069976 +0000 UTC m=+17.990932446, 105.389736ms after stability testing.go:149: cluster: 2025-02-09T16:52:23.660Z [INFO] stable state for Follower reached at 2025-02-09 16:52:23.555310022 +0000 UTC m=+17.885542710 (2 nodes), 54.980246ms from start of poll, 55.317653ms from cluster start. Timeout at 2025-02-09 16:52:23.66069976 +0000 UTC m=+17.990932446, 105.389736ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.660Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:23.660Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.765Z [INFO] stable state for Leader reached at 2025-02-09 16:52:23.66083766 +0000 UTC m=+17.991070346 (1 nodes), 1.111µs from start of poll, 160.845289ms from cluster start. Timeout at 2025-02-09 16:52:23.765845539 +0000 UTC m=+18.096078227, 105.007881ms after stability testing.go:149: cluster: 2025-02-09T16:52:23.765Z [INFO] stable state for Leader reached at 2025-02-09 16:52:23.66083766 +0000 UTC m=+17.991070346 (1 nodes), 1.111µs from start of poll, 160.845289ms from cluster start. Timeout at 2025-02-09 16:52:23.765845539 +0000 UTC m=+18.096078227, 105.007881ms after stability testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.766Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: aborting pipeline replication: peer="{Voter server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f e87cd036-16ed-bbe5-c7b4-b2555fd66b3f}" testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.766Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: aborting pipeline replication: peer="{Voter server-e87cd036-16ed-bbe5-c7b4-b2555fd66b3f e87cd036-16ed-bbe5-c7b4-b2555fd66b3f}" testLoggerAdapter verbose: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.766Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: aborting pipeline replication: peer="{Voter server-7b71f52c-6c43-55cd-99e7-25a5c158e819 7b71f52c-6c43-55cd-99e7-25a5c158e819}" testing.go:149: server-65b2dac5-96e3-ea00-e0af-1b2edb929829: 2025-02-09T16:52:23.766Z [INFO] server-65b2dac5-96e3-ea00-e0af-1b2edb929829: aborting pipeline replication: peer="{Voter server-7b71f52c-6c43-55cd-99e7-25a5c158e819 7b71f52c-6c43-55cd-99e7-25a5c158e819}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) === RUN TestRaft_LeadershipTransferToBehindServer testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:23.766Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16e8b072-5145-b415-53d5-1de4318fabe9 Address:16e8b072-5145-b415-53d5-1de4318fabe9} {Suffrage:Voter ID:server-b1c3ed07-44db-be29-dee9-45cd87462e5c Address:b1c3ed07-44db-be29-dee9-45cd87462e5c} {Suffrage:Voter ID:server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 Address:9886f2fc-315e-8a1a-ea34-63a0c4d99af6}]" testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:23.766Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16e8b072-5145-b415-53d5-1de4318fabe9 Address:16e8b072-5145-b415-53d5-1de4318fabe9} {Suffrage:Voter ID:server-b1c3ed07-44db-be29-dee9-45cd87462e5c Address:b1c3ed07-44db-be29-dee9-45cd87462e5c} {Suffrage:Voter ID:server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 Address:9886f2fc-315e-8a1a-ea34-63a0c4d99af6}]" testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:23.767Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: entering follower state: follower="Node at 16e8b072-5145-b415-53d5-1de4318fabe9 [Follower]" leader= testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:23.767Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: entering follower state: follower="Node at 16e8b072-5145-b415-53d5-1de4318fabe9 [Follower]" leader= testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.767Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16e8b072-5145-b415-53d5-1de4318fabe9 Address:16e8b072-5145-b415-53d5-1de4318fabe9} {Suffrage:Voter ID:server-b1c3ed07-44db-be29-dee9-45cd87462e5c Address:b1c3ed07-44db-be29-dee9-45cd87462e5c} {Suffrage:Voter ID:server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 Address:9886f2fc-315e-8a1a-ea34-63a0c4d99af6}]" testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.767Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16e8b072-5145-b415-53d5-1de4318fabe9 Address:16e8b072-5145-b415-53d5-1de4318fabe9} {Suffrage:Voter ID:server-b1c3ed07-44db-be29-dee9-45cd87462e5c Address:b1c3ed07-44db-be29-dee9-45cd87462e5c} {Suffrage:Voter ID:server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 Address:9886f2fc-315e-8a1a-ea34-63a0c4d99af6}]" testLoggerAdapter verbose: server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6: 2025-02-09T16:52:23.767Z [INFO] server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16e8b072-5145-b415-53d5-1de4318fabe9 Address:16e8b072-5145-b415-53d5-1de4318fabe9} {Suffrage:Voter ID:server-b1c3ed07-44db-be29-dee9-45cd87462e5c Address:b1c3ed07-44db-be29-dee9-45cd87462e5c} {Suffrage:Voter ID:server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 Address:9886f2fc-315e-8a1a-ea34-63a0c4d99af6}]" testing.go:149: server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6: 2025-02-09T16:52:23.767Z [INFO] server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16e8b072-5145-b415-53d5-1de4318fabe9 Address:16e8b072-5145-b415-53d5-1de4318fabe9} {Suffrage:Voter ID:server-b1c3ed07-44db-be29-dee9-45cd87462e5c Address:b1c3ed07-44db-be29-dee9-45cd87462e5c} {Suffrage:Voter ID:server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 Address:9886f2fc-315e-8a1a-ea34-63a0c4d99af6}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.767Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:23.767Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.767Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: entering follower state: follower="Node at b1c3ed07-44db-be29-dee9-45cd87462e5c [Follower]" leader= testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.767Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: entering follower state: follower="Node at b1c3ed07-44db-be29-dee9-45cd87462e5c [Follower]" leader= testLoggerAdapter verbose: server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6: 2025-02-09T16:52:23.767Z [INFO] server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6: entering follower state: follower="Node at 9886f2fc-315e-8a1a-ea34-63a0c4d99af6 [Follower]" leader= testing.go:149: server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6: 2025-02-09T16:52:23.767Z [INFO] server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6: entering follower state: follower="Node at 9886f2fc-315e-8a1a-ea34-63a0c4d99af6 [Follower]" leader= testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [WARN] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [WARN] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: entering candidate state: node="Node at b1c3ed07-44db-be29-dee9-45cd87462e5c [Candidate]" term=2 testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: entering candidate state: node="Node at b1c3ed07-44db-be29-dee9-45cd87462e5c [Candidate]" term=2 testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: election won: tally=2 testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: election won: tally=2 testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: entering leader state: leader="Node at b1c3ed07-44db-be29-dee9-45cd87462e5c [Leader]" testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: entering leader state: leader="Node at b1c3ed07-44db-be29-dee9-45cd87462e5c [Leader]" testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: added peer, starting replication: peer=server-16e8b072-5145-b415-53d5-1de4318fabe9 testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: added peer, starting replication: peer=server-16e8b072-5145-b415-53d5-1de4318fabe9 testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: added peer, starting replication: peer=server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: added peer, starting replication: peer=server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: pipelining replication: peer="{Voter server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 9886f2fc-315e-8a1a-ea34-63a0c4d99af6}" testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: pipelining replication: peer="{Voter server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 9886f2fc-315e-8a1a-ea34-63a0c4d99af6}" testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: pipelining replication: peer="{Voter server-16e8b072-5145-b415-53d5-1de4318fabe9 16e8b072-5145-b415-53d5-1de4318fabe9}" testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:23.839Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: pipelining replication: peer="{Voter server-16e8b072-5145-b415-53d5-1de4318fabe9 16e8b072-5145-b415-53d5-1de4318fabe9}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.945Z [INFO] stable state for Leader reached at 2025-02-09 16:52:23.839838404 +0000 UTC m=+18.170071096 (1 nodes), 72.499296ms from start of poll, 73.017335ms from cluster start. Timeout at 2025-02-09 16:52:23.945432827 +0000 UTC m=+18.275665513, 105.594417ms after stability testing.go:149: cluster: 2025-02-09T16:52:23.945Z [INFO] stable state for Leader reached at 2025-02-09 16:52:23.839838404 +0000 UTC m=+18.170071096 (1 nodes), 72.499296ms from start of poll, 73.017335ms from cluster start. Timeout at 2025-02-09 16:52:23.945432827 +0000 UTC m=+18.275665513, 105.594417ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:23.945Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:23.945Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:24.050Z [INFO] stable state for Follower reached at 2025-02-09 16:52:23.945593691 +0000 UTC m=+18.275826378 (2 nodes), 1.057µs from start of poll, 178.772617ms from cluster start. Timeout at 2025-02-09 16:52:24.050762378 +0000 UTC m=+18.380995066, 105.168688ms after stability testing.go:149: cluster: 2025-02-09T16:52:24.050Z [INFO] stable state for Follower reached at 2025-02-09 16:52:23.945593691 +0000 UTC m=+18.275826378 (2 nodes), 1.057µs from start of poll, 178.772617ms from cluster start. Timeout at 2025-02-09 16:52:24.050762378 +0000 UTC m=+18.380995066, 105.168688ms after stability testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.078Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: entering candidate state: node="Node at 16e8b072-5145-b415-53d5-1de4318fabe9 [Candidate]" term=3 testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.078Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: entering candidate state: node="Node at 16e8b072-5145-b415-53d5-1de4318fabe9 [Candidate]" term=3 testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: election won: tally=2 testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: election won: tally=2 testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: entering leader state: leader="Node at 16e8b072-5145-b415-53d5-1de4318fabe9 [Leader]" testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: entering leader state: leader="Node at 16e8b072-5145-b415-53d5-1de4318fabe9 [Leader]" testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: added peer, starting replication: peer=server-b1c3ed07-44db-be29-dee9-45cd87462e5c testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: added peer, starting replication: peer=server-b1c3ed07-44db-be29-dee9-45cd87462e5c testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:24.079Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: entering follower state: follower="Node at b1c3ed07-44db-be29-dee9-45cd87462e5c [Follower]" leader=16e8b072-5145-b415-53d5-1de4318fabe9 testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:24.079Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: entering follower state: follower="Node at b1c3ed07-44db-be29-dee9-45cd87462e5c [Follower]" leader=16e8b072-5145-b415-53d5-1de4318fabe9 testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:24.079Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: aborting pipeline replication: peer="{Voter server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 9886f2fc-315e-8a1a-ea34-63a0c4d99af6}" testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:24.079Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: aborting pipeline replication: peer="{Voter server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 9886f2fc-315e-8a1a-ea34-63a0c4d99af6}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:24.079Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:24.079Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:24.079Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: aborting pipeline replication: peer="{Voter server-16e8b072-5145-b415-53d5-1de4318fabe9 16e8b072-5145-b415-53d5-1de4318fabe9}" testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:24.079Z [INFO] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: aborting pipeline replication: peer="{Voter server-16e8b072-5145-b415-53d5-1de4318fabe9 16e8b072-5145-b415-53d5-1de4318fabe9}" testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: added peer, starting replication: peer=server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: added peer, starting replication: peer=server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: pipelining replication: peer="{Voter server-b1c3ed07-44db-be29-dee9-45cd87462e5c b1c3ed07-44db-be29-dee9-45cd87462e5c}" testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: pipelining replication: peer="{Voter server-b1c3ed07-44db-be29-dee9-45cd87462e5c b1c3ed07-44db-be29-dee9-45cd87462e5c}" testLoggerAdapter verbose: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:24.079Z [WARN] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: rejecting vote request since our last term is greater: candidate=16e8b072-5145-b415-53d5-1de4318fabe9 last-term=3 last-candidate-term=2 testing.go:149: server-b1c3ed07-44db-be29-dee9-45cd87462e5c: 2025-02-09T16:52:24.079Z [WARN] server-b1c3ed07-44db-be29-dee9-45cd87462e5c: rejecting vote request since our last term is greater: candidate=16e8b072-5145-b415-53d5-1de4318fabe9 last-term=3 last-candidate-term=2 testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: pipelining replication: peer="{Voter server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 9886f2fc-315e-8a1a-ea34-63a0c4d99af6}" testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.079Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: pipelining replication: peer="{Voter server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 9886f2fc-315e-8a1a-ea34-63a0c4d99af6}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:24.185Z [INFO] stable state for Leader reached at 2025-02-09 16:52:24.079902814 +0000 UTC m=+18.410135503 (1 nodes), 383.14µs from start of poll, 313.081742ms from cluster start. Timeout at 2025-02-09 16:52:24.185240205 +0000 UTC m=+18.515472892, 105.337389ms after stability testing.go:149: cluster: 2025-02-09T16:52:24.185Z [INFO] stable state for Leader reached at 2025-02-09 16:52:24.079902814 +0000 UTC m=+18.410135503 (1 nodes), 383.14µs from start of poll, 313.081742ms from cluster start. Timeout at 2025-02-09 16:52:24.185240205 +0000 UTC m=+18.515472892, 105.337389ms after stability testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.185Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: aborting pipeline replication: peer="{Voter server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 9886f2fc-315e-8a1a-ea34-63a0c4d99af6}" testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.185Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: aborting pipeline replication: peer="{Voter server-9886f2fc-315e-8a1a-ea34-63a0c4d99af6 9886f2fc-315e-8a1a-ea34-63a0c4d99af6}" testLoggerAdapter verbose: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.185Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: aborting pipeline replication: peer="{Voter server-b1c3ed07-44db-be29-dee9-45cd87462e5c b1c3ed07-44db-be29-dee9-45cd87462e5c}" testing.go:149: server-16e8b072-5145-b415-53d5-1de4318fabe9: 2025-02-09T16:52:24.185Z [INFO] server-16e8b072-5145-b415-53d5-1de4318fabe9: aborting pipeline replication: peer="{Voter server-b1c3ed07-44db-be29-dee9-45cd87462e5c b1c3ed07-44db-be29-dee9-45cd87462e5c}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s) === RUN TestRaft_LeadershipTransferToItself testLoggerAdapter verbose: server-498929cb-3dd0-d9bd-08bd-a53d623cd35f: 2025-02-09T16:52:24.186Z [INFO] server-498929cb-3dd0-d9bd-08bd-a53d623cd35f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-498929cb-3dd0-d9bd-08bd-a53d623cd35f Address:498929cb-3dd0-d9bd-08bd-a53d623cd35f} {Suffrage:Voter ID:server-3fa5d42b-6b53-886b-8857-bb0eae8f716a Address:3fa5d42b-6b53-886b-8857-bb0eae8f716a} {Suffrage:Voter ID:server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b Address:74d7c61b-5e03-4b6b-9ea3-196e7c77f44b}]" testing.go:149: server-498929cb-3dd0-d9bd-08bd-a53d623cd35f: 2025-02-09T16:52:24.186Z [INFO] server-498929cb-3dd0-d9bd-08bd-a53d623cd35f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-498929cb-3dd0-d9bd-08bd-a53d623cd35f Address:498929cb-3dd0-d9bd-08bd-a53d623cd35f} {Suffrage:Voter ID:server-3fa5d42b-6b53-886b-8857-bb0eae8f716a Address:3fa5d42b-6b53-886b-8857-bb0eae8f716a} {Suffrage:Voter ID:server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b Address:74d7c61b-5e03-4b6b-9ea3-196e7c77f44b}]" testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.186Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-498929cb-3dd0-d9bd-08bd-a53d623cd35f Address:498929cb-3dd0-d9bd-08bd-a53d623cd35f} {Suffrage:Voter ID:server-3fa5d42b-6b53-886b-8857-bb0eae8f716a Address:3fa5d42b-6b53-886b-8857-bb0eae8f716a} {Suffrage:Voter ID:server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b Address:74d7c61b-5e03-4b6b-9ea3-196e7c77f44b}]" testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.186Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-498929cb-3dd0-d9bd-08bd-a53d623cd35f Address:498929cb-3dd0-d9bd-08bd-a53d623cd35f} {Suffrage:Voter ID:server-3fa5d42b-6b53-886b-8857-bb0eae8f716a Address:3fa5d42b-6b53-886b-8857-bb0eae8f716a} {Suffrage:Voter ID:server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b Address:74d7c61b-5e03-4b6b-9ea3-196e7c77f44b}]" testLoggerAdapter verbose: server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b: 2025-02-09T16:52:24.186Z [INFO] server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-498929cb-3dd0-d9bd-08bd-a53d623cd35f Address:498929cb-3dd0-d9bd-08bd-a53d623cd35f} {Suffrage:Voter ID:server-3fa5d42b-6b53-886b-8857-bb0eae8f716a Address:3fa5d42b-6b53-886b-8857-bb0eae8f716a} {Suffrage:Voter ID:server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b Address:74d7c61b-5e03-4b6b-9ea3-196e7c77f44b}]" testing.go:149: server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b: 2025-02-09T16:52:24.186Z [INFO] server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-498929cb-3dd0-d9bd-08bd-a53d623cd35f Address:498929cb-3dd0-d9bd-08bd-a53d623cd35f} {Suffrage:Voter ID:server-3fa5d42b-6b53-886b-8857-bb0eae8f716a Address:3fa5d42b-6b53-886b-8857-bb0eae8f716a} {Suffrage:Voter ID:server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b Address:74d7c61b-5e03-4b6b-9ea3-196e7c77f44b}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:24.186Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:24.186Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-498929cb-3dd0-d9bd-08bd-a53d623cd35f: 2025-02-09T16:52:24.186Z [INFO] server-498929cb-3dd0-d9bd-08bd-a53d623cd35f: entering follower state: follower="Node at 498929cb-3dd0-d9bd-08bd-a53d623cd35f [Follower]" leader= testing.go:149: server-498929cb-3dd0-d9bd-08bd-a53d623cd35f: 2025-02-09T16:52:24.186Z [INFO] server-498929cb-3dd0-d9bd-08bd-a53d623cd35f: entering follower state: follower="Node at 498929cb-3dd0-d9bd-08bd-a53d623cd35f [Follower]" leader= testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.186Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: entering follower state: follower="Node at 3fa5d42b-6b53-886b-8857-bb0eae8f716a [Follower]" leader= testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.186Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: entering follower state: follower="Node at 3fa5d42b-6b53-886b-8857-bb0eae8f716a [Follower]" leader= testLoggerAdapter verbose: server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b: 2025-02-09T16:52:24.187Z [INFO] server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b: entering follower state: follower="Node at 74d7c61b-5e03-4b6b-9ea3-196e7c77f44b [Follower]" leader= testing.go:149: server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b: 2025-02-09T16:52:24.187Z [INFO] server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b: entering follower state: follower="Node at 74d7c61b-5e03-4b6b-9ea3-196e7c77f44b [Follower]" leader= testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [WARN] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [WARN] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: entering candidate state: node="Node at 3fa5d42b-6b53-886b-8857-bb0eae8f716a [Candidate]" term=2 testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: entering candidate state: node="Node at 3fa5d42b-6b53-886b-8857-bb0eae8f716a [Candidate]" term=2 testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: election won: tally=2 testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: election won: tally=2 testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: entering leader state: leader="Node at 3fa5d42b-6b53-886b-8857-bb0eae8f716a [Leader]" testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: entering leader state: leader="Node at 3fa5d42b-6b53-886b-8857-bb0eae8f716a [Leader]" testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: added peer, starting replication: peer=server-498929cb-3dd0-d9bd-08bd-a53d623cd35f testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: added peer, starting replication: peer=server-498929cb-3dd0-d9bd-08bd-a53d623cd35f testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: added peer, starting replication: peer=server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.272Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: added peer, starting replication: peer=server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.273Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: pipelining replication: peer="{Voter server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b 74d7c61b-5e03-4b6b-9ea3-196e7c77f44b}" testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.273Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: pipelining replication: peer="{Voter server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b 74d7c61b-5e03-4b6b-9ea3-196e7c77f44b}" testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.273Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: pipelining replication: peer="{Voter server-498929cb-3dd0-d9bd-08bd-a53d623cd35f 498929cb-3dd0-d9bd-08bd-a53d623cd35f}" testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.273Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: pipelining replication: peer="{Voter server-498929cb-3dd0-d9bd-08bd-a53d623cd35f 498929cb-3dd0-d9bd-08bd-a53d623cd35f}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:24.378Z [INFO] stable state for Leader reached at 2025-02-09 16:52:24.273294041 +0000 UTC m=+18.603526729 (1 nodes), 86.423773ms from start of poll, 86.899185ms from cluster start. Timeout at 2025-02-09 16:52:24.37855957 +0000 UTC m=+18.708792257, 105.265528ms after stability testing.go:149: cluster: 2025-02-09T16:52:24.378Z [INFO] stable state for Leader reached at 2025-02-09 16:52:24.273294041 +0000 UTC m=+18.603526729 (1 nodes), 86.423773ms from start of poll, 86.899185ms from cluster start. Timeout at 2025-02-09 16:52:24.37855957 +0000 UTC m=+18.708792257, 105.265528ms after stability testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.378Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: cannot transfer leadership to itself testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.378Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: cannot transfer leadership to itself testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.378Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: aborting pipeline replication: peer="{Voter server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b 74d7c61b-5e03-4b6b-9ea3-196e7c77f44b}" testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.378Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: aborting pipeline replication: peer="{Voter server-74d7c61b-5e03-4b6b-9ea3-196e7c77f44b 74d7c61b-5e03-4b6b-9ea3-196e7c77f44b}" testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.378Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: aborting pipeline replication: peer="{Voter server-498929cb-3dd0-d9bd-08bd-a53d623cd35f 498929cb-3dd0-d9bd-08bd-a53d623cd35f}" testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.378Z [INFO] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: aborting pipeline replication: peer="{Voter server-498929cb-3dd0-d9bd-08bd-a53d623cd35f 498929cb-3dd0-d9bd-08bd-a53d623cd35f}" testLoggerAdapter verbose: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.429Z [ERROR] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: failed to heartbeat to: peer=498929cb-3dd0-d9bd-08bd-a53d623cd35f error="command timed out" testing.go:149: server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: 2025-02-09T16:52:24.429Z [ERROR] server-3fa5d42b-6b53-886b-8857-bb0eae8f716a: failed to heartbeat to: peer=498929cb-3dd0-d9bd-08bd-a53d623cd35f error="command timed out" --- PASS: TestRaft_LeadershipTransferToItself (0.24s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout testLoggerAdapter verbose: server-920173a8-7fda-4235-7a39-d947808e55b5: 2025-02-09T16:52:24.430Z [INFO] server-920173a8-7fda-4235-7a39-d947808e55b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-920173a8-7fda-4235-7a39-d947808e55b5 Address:920173a8-7fda-4235-7a39-d947808e55b5} {Suffrage:Voter ID:server-96a2e38d-7567-c31d-918f-35cfd74c62aa Address:96a2e38d-7567-c31d-918f-35cfd74c62aa} {Suffrage:Voter ID:server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 Address:21c74c25-18fc-0c7b-81d3-e0432680eeb2}]" testing.go:149: server-920173a8-7fda-4235-7a39-d947808e55b5: 2025-02-09T16:52:24.430Z [INFO] server-920173a8-7fda-4235-7a39-d947808e55b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-920173a8-7fda-4235-7a39-d947808e55b5 Address:920173a8-7fda-4235-7a39-d947808e55b5} {Suffrage:Voter ID:server-96a2e38d-7567-c31d-918f-35cfd74c62aa Address:96a2e38d-7567-c31d-918f-35cfd74c62aa} {Suffrage:Voter ID:server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 Address:21c74c25-18fc-0c7b-81d3-e0432680eeb2}]" testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.430Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-920173a8-7fda-4235-7a39-d947808e55b5 Address:920173a8-7fda-4235-7a39-d947808e55b5} {Suffrage:Voter ID:server-96a2e38d-7567-c31d-918f-35cfd74c62aa Address:96a2e38d-7567-c31d-918f-35cfd74c62aa} {Suffrage:Voter ID:server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 Address:21c74c25-18fc-0c7b-81d3-e0432680eeb2}]" testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.430Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-920173a8-7fda-4235-7a39-d947808e55b5 Address:920173a8-7fda-4235-7a39-d947808e55b5} {Suffrage:Voter ID:server-96a2e38d-7567-c31d-918f-35cfd74c62aa Address:96a2e38d-7567-c31d-918f-35cfd74c62aa} {Suffrage:Voter ID:server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 Address:21c74c25-18fc-0c7b-81d3-e0432680eeb2}]" testLoggerAdapter verbose: server-21c74c25-18fc-0c7b-81d3-e0432680eeb2: 2025-02-09T16:52:24.430Z [INFO] server-21c74c25-18fc-0c7b-81d3-e0432680eeb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-920173a8-7fda-4235-7a39-d947808e55b5 Address:920173a8-7fda-4235-7a39-d947808e55b5} {Suffrage:Voter ID:server-96a2e38d-7567-c31d-918f-35cfd74c62aa Address:96a2e38d-7567-c31d-918f-35cfd74c62aa} {Suffrage:Voter ID:server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 Address:21c74c25-18fc-0c7b-81d3-e0432680eeb2}]" testing.go:149: server-21c74c25-18fc-0c7b-81d3-e0432680eeb2: 2025-02-09T16:52:24.430Z [INFO] server-21c74c25-18fc-0c7b-81d3-e0432680eeb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-920173a8-7fda-4235-7a39-d947808e55b5 Address:920173a8-7fda-4235-7a39-d947808e55b5} {Suffrage:Voter ID:server-96a2e38d-7567-c31d-918f-35cfd74c62aa Address:96a2e38d-7567-c31d-918f-35cfd74c62aa} {Suffrage:Voter ID:server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 Address:21c74c25-18fc-0c7b-81d3-e0432680eeb2}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:24.430Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:24.430Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-920173a8-7fda-4235-7a39-d947808e55b5: 2025-02-09T16:52:24.430Z [INFO] server-920173a8-7fda-4235-7a39-d947808e55b5: entering follower state: follower="Node at 920173a8-7fda-4235-7a39-d947808e55b5 [Follower]" leader= testing.go:149: server-920173a8-7fda-4235-7a39-d947808e55b5: 2025-02-09T16:52:24.430Z [INFO] server-920173a8-7fda-4235-7a39-d947808e55b5: entering follower state: follower="Node at 920173a8-7fda-4235-7a39-d947808e55b5 [Follower]" leader= testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.430Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: entering follower state: follower="Node at 96a2e38d-7567-c31d-918f-35cfd74c62aa [Follower]" leader= testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.430Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: entering follower state: follower="Node at 96a2e38d-7567-c31d-918f-35cfd74c62aa [Follower]" leader= testLoggerAdapter verbose: server-21c74c25-18fc-0c7b-81d3-e0432680eeb2: 2025-02-09T16:52:24.430Z [INFO] server-21c74c25-18fc-0c7b-81d3-e0432680eeb2: entering follower state: follower="Node at 21c74c25-18fc-0c7b-81d3-e0432680eeb2 [Follower]" leader= testing.go:149: server-21c74c25-18fc-0c7b-81d3-e0432680eeb2: 2025-02-09T16:52:24.430Z [INFO] server-21c74c25-18fc-0c7b-81d3-e0432680eeb2: entering follower state: follower="Node at 21c74c25-18fc-0c7b-81d3-e0432680eeb2 [Follower]" leader= testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [WARN] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [WARN] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: entering candidate state: node="Node at 96a2e38d-7567-c31d-918f-35cfd74c62aa [Candidate]" term=2 testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: entering candidate state: node="Node at 96a2e38d-7567-c31d-918f-35cfd74c62aa [Candidate]" term=2 testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: election won: tally=2 testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: election won: tally=2 testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: entering leader state: leader="Node at 96a2e38d-7567-c31d-918f-35cfd74c62aa [Leader]" testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: entering leader state: leader="Node at 96a2e38d-7567-c31d-918f-35cfd74c62aa [Leader]" testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: added peer, starting replication: peer=server-920173a8-7fda-4235-7a39-d947808e55b5 testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: added peer, starting replication: peer=server-920173a8-7fda-4235-7a39-d947808e55b5 testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: added peer, starting replication: peer=server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: added peer, starting replication: peer=server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: pipelining replication: peer="{Voter server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 21c74c25-18fc-0c7b-81d3-e0432680eeb2}" testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: pipelining replication: peer="{Voter server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 21c74c25-18fc-0c7b-81d3-e0432680eeb2}" testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: pipelining replication: peer="{Voter server-920173a8-7fda-4235-7a39-d947808e55b5 920173a8-7fda-4235-7a39-d947808e55b5}" testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.518Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: pipelining replication: peer="{Voter server-920173a8-7fda-4235-7a39-d947808e55b5 920173a8-7fda-4235-7a39-d947808e55b5}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:24.624Z [INFO] stable state for Leader reached at 2025-02-09 16:52:24.518989253 +0000 UTC m=+18.849221941 (1 nodes), 88.140567ms from start of poll, 88.584761ms from cluster start. Timeout at 2025-02-09 16:52:24.624041826 +0000 UTC m=+18.954274517, 105.052576ms after stability testing.go:149: cluster: 2025-02-09T16:52:24.624Z [INFO] stable state for Leader reached at 2025-02-09 16:52:24.518989253 +0000 UTC m=+18.849221941 (1 nodes), 88.140567ms from start of poll, 88.584761ms from cluster start. Timeout at 2025-02-09 16:52:24.624041826 +0000 UTC m=+18.954274517, 105.052576ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:24.624Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:24.624Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T16:52:24.729Z [INFO] stable state for Follower reached at 2025-02-09 16:52:24.62417429 +0000 UTC m=+18.954406975 (2 nodes), 1.006µs from start of poll, 193.769795ms from cluster start. Timeout at 2025-02-09 16:52:24.729542288 +0000 UTC m=+19.059774975, 105.368ms after stability testing.go:149: cluster: 2025-02-09T16:52:24.729Z [INFO] stable state for Follower reached at 2025-02-09 16:52:24.62417429 +0000 UTC m=+18.954406975 (2 nodes), 1.006µs from start of poll, 193.769795ms from cluster start. Timeout at 2025-02-09 16:52:24.729542288 +0000 UTC m=+19.059774975, 105.368ms after stability testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.974Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: aborting pipeline replication: peer="{Voter server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 21c74c25-18fc-0c7b-81d3-e0432680eeb2}" testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.974Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: aborting pipeline replication: peer="{Voter server-21c74c25-18fc-0c7b-81d3-e0432680eeb2 21c74c25-18fc-0c7b-81d3-e0432680eeb2}" testLoggerAdapter verbose: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.974Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: aborting pipeline replication: peer="{Voter server-920173a8-7fda-4235-7a39-d947808e55b5 920173a8-7fda-4235-7a39-d947808e55b5}" testing.go:149: server-96a2e38d-7567-c31d-918f-35cfd74c62aa: 2025-02-09T16:52:24.974Z [INFO] server-96a2e38d-7567-c31d-918f-35cfd74c62aa: aborting pipeline replication: peer="{Voter server-920173a8-7fda-4235-7a39-d947808e55b5 920173a8-7fda-4235-7a39-d947808e55b5}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.55s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap testLoggerAdapter verbose: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:24.975Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e002409a-2eae-e863-14de-82de774e182d Address:e002409a-2eae-e863-14de-82de774e182d} {Suffrage:Voter ID:server-24e230a7-ff12-1719-381d-ae2a73edb0d0 Address:24e230a7-ff12-1719-381d-ae2a73edb0d0}]" testing.go:149: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:24.975Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e002409a-2eae-e863-14de-82de774e182d Address:e002409a-2eae-e863-14de-82de774e182d} {Suffrage:Voter ID:server-24e230a7-ff12-1719-381d-ae2a73edb0d0 Address:24e230a7-ff12-1719-381d-ae2a73edb0d0}]" testLoggerAdapter verbose: server-24e230a7-ff12-1719-381d-ae2a73edb0d0: 2025-02-09T16:52:24.975Z [INFO] server-24e230a7-ff12-1719-381d-ae2a73edb0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e002409a-2eae-e863-14de-82de774e182d Address:e002409a-2eae-e863-14de-82de774e182d} {Suffrage:Voter ID:server-24e230a7-ff12-1719-381d-ae2a73edb0d0 Address:24e230a7-ff12-1719-381d-ae2a73edb0d0}]" testing.go:149: server-24e230a7-ff12-1719-381d-ae2a73edb0d0: 2025-02-09T16:52:24.975Z [INFO] server-24e230a7-ff12-1719-381d-ae2a73edb0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e002409a-2eae-e863-14de-82de774e182d Address:e002409a-2eae-e863-14de-82de774e182d} {Suffrage:Voter ID:server-24e230a7-ff12-1719-381d-ae2a73edb0d0 Address:24e230a7-ff12-1719-381d-ae2a73edb0d0}]" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:24.975Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T16:52:24.975Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:24.975Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: entering follower state: follower="Node at e002409a-2eae-e863-14de-82de774e182d [Follower]" leader= testing.go:149: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:24.975Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: entering follower state: follower="Node at e002409a-2eae-e863-14de-82de774e182d [Follower]" leader= testLoggerAdapter verbose: server-24e230a7-ff12-1719-381d-ae2a73edb0d0: 2025-02-09T16:52:24.975Z [INFO] server-24e230a7-ff12-1719-381d-ae2a73edb0d0: entering follower state: follower="Node at 24e230a7-ff12-1719-381d-ae2a73edb0d0 [Follower]" leader= testing.go:149: server-24e230a7-ff12-1719-381d-ae2a73edb0d0: 2025-02-09T16:52:24.975Z [INFO] server-24e230a7-ff12-1719-381d-ae2a73edb0d0: entering follower state: follower="Node at 24e230a7-ff12-1719-381d-ae2a73edb0d0 [Follower]" leader= testLoggerAdapter verbose: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [WARN] server-e002409a-2eae-e863-14de-82de774e182d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [WARN] server-e002409a-2eae-e863-14de-82de774e182d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: entering candidate state: node="Node at e002409a-2eae-e863-14de-82de774e182d [Candidate]" term=2 testing.go:149: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: entering candidate state: node="Node at e002409a-2eae-e863-14de-82de774e182d [Candidate]" term=2 testLoggerAdapter verbose: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: election won: tally=2 testing.go:149: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: election won: tally=2 testLoggerAdapter verbose: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: entering leader state: leader="Node at e002409a-2eae-e863-14de-82de774e182d [Leader]" testing.go:149: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: entering leader state: leader="Node at e002409a-2eae-e863-14de-82de774e182d [Leader]" testLoggerAdapter verbose: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: added peer, starting replication: peer=server-24e230a7-ff12-1719-381d-ae2a73edb0d0 testing.go:149: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: added peer, starting replication: peer=server-24e230a7-ff12-1719-381d-ae2a73edb0d0 testLoggerAdapter verbose: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: pipelining replication: peer="{Voter server-24e230a7-ff12-1719-381d-ae2a73edb0d0 24e230a7-ff12-1719-381d-ae2a73edb0d0}" testing.go:149: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.043Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: pipelining replication: peer="{Voter server-24e230a7-ff12-1719-381d-ae2a73edb0d0 24e230a7-ff12-1719-381d-ae2a73edb0d0}" testLoggerAdapter verbose: cluster: 2025-02-09T16:52:25.149Z [INFO] stable state for Follower reached at 2025-02-09 16:52:25.043502237 +0000 UTC m=+19.373734925 (1 nodes), 67.713002ms from start of poll, 68.004739ms from cluster start. Timeout at 2025-02-09 16:52:25.149311304 +0000 UTC m=+19.479544002, 105.809077ms after stability testing.go:149: cluster: 2025-02-09T16:52:25.149Z [INFO] stable state for Follower reached at 2025-02-09 16:52:25.043502237 +0000 UTC m=+19.373734925 (1 nodes), 67.713002ms from start of poll, 68.004739ms from cluster start. Timeout at 2025-02-09 16:52:25.149311304 +0000 UTC m=+19.479544002, 105.809077ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T16:52:25.149Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T16:52:25.149Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T16:52:25.255Z [INFO] stable state for Leader reached at 2025-02-09 16:52:25.149452162 +0000 UTC m=+19.479684855 (1 nodes), 838ns from start of poll, 173.954669ms from cluster start. Timeout at 2025-02-09 16:52:25.25509387 +0000 UTC m=+19.585326556, 105.641701ms after stability testing.go:149: cluster: 2025-02-09T16:52:25.255Z [INFO] stable state for Leader reached at 2025-02-09 16:52:25.149452162 +0000 UTC m=+19.479684855 (1 nodes), 838ns from start of poll, 173.954669ms from cluster start. Timeout at 2025-02-09 16:52:25.25509387 +0000 UTC m=+19.585326556, 105.641701ms after stability testLoggerAdapter verbose: server-24e230a7-ff12-1719-381d-ae2a73edb0d0: 2025-02-09T16:52:25.266Z [INFO] server-24e230a7-ff12-1719-381d-ae2a73edb0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e002409a-2eae-e863-14de-82de774e182d Address:e002409a-2eae-e863-14de-82de774e182d} {Suffrage:Voter ID:server-24e230a7-ff12-1719-381d-ae2a73edb0d0 Address:24e230a7-ff12-1719-381d-ae2a73edb0d0}]" testing.go:149: server-24e230a7-ff12-1719-381d-ae2a73edb0d0: 2025-02-09T16:52:25.266Z [INFO] server-24e230a7-ff12-1719-381d-ae2a73edb0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e002409a-2eae-e863-14de-82de774e182d Address:e002409a-2eae-e863-14de-82de774e182d} {Suffrage:Voter ID:server-24e230a7-ff12-1719-381d-ae2a73edb0d0 Address:24e230a7-ff12-1719-381d-ae2a73edb0d0}]" testLoggerAdapter verbose: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.266Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: aborting pipeline replication: peer="{Voter server-24e230a7-ff12-1719-381d-ae2a73edb0d0 24e230a7-ff12-1719-381d-ae2a73edb0d0}" testing.go:149: server-e002409a-2eae-e863-14de-82de774e182d: 2025-02-09T16:52:25.266Z [INFO] server-e002409a-2eae-e863-14de-82de774e182d: aborting pipeline replication: peer="{Voter server-24e230a7-ff12-1719-381d-ae2a73edb0d0 24e230a7-ff12-1719-381d-ae2a73edb0d0}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.29s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS ok github.com/hashicorp/raft 19.611s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary dh binary --buildsystem=golang --with=golang dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installinit -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dpkg-gencontrol: warning: Depends field of package golang-github-hashicorp-raft-dev: substitution variable ${shlibs:Depends} used, but is not defined dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.1.2-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.1.2-1_amd64.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.1.2-1_amd64.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/2584380 and its subdirectories I: Current time: Sun Feb 9 04:52:31 -12 2025 I: pbuilder-time-stamp: 1739119951 Mon Jan 8 10:29:24 UTC 2024 I: 1st build successful. Starting 2nd build on remote node ionos1-amd64.debian.net. Mon Jan 8 10:29:24 UTC 2024 I: Preparing to do remote build '2' on ionos1-amd64.debian.net. Mon Jan 8 10:39:21 UTC 2024 I: Deleting $TMPDIR on ionos1-amd64.debian.net. Mon Jan 8 10:39:22 UTC 2024 I: golang-github-hashicorp-raft_1.1.2-1_amd64.changes: Format: 1.8 Date: Sun, 19 Jan 2020 18:15:42 +1100 Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.1.2-1 Distribution: unstable Urgency: medium Maintainer: Debian Go Packaging Team Changed-By: Dmitry Smirnov Description: golang-github-hashicorp-raft-dev - implementation of the Raft consensus protocol Changes: golang-github-hashicorp-raft (1.1.2-1) unstable; urgency=medium . * New upstream release. Checksums-Sha1: 4fcae6db8ecb88479bbb266ab817605e714c9d9e 90036 golang-github-hashicorp-raft-dev_1.1.2-1_all.deb 99e6615daad32c3a4b58789ff7be4ded060b0065 8301 golang-github-hashicorp-raft_1.1.2-1_amd64.buildinfo Checksums-Sha256: 579a17edb3722463f068b5ad1c77fca12707493b919392d4cb32cd828d42473e 90036 golang-github-hashicorp-raft-dev_1.1.2-1_all.deb 2c2c78110e0f377b2ebcec725248ad5ea28cbc488276a5323fe68977c523162a 8301 golang-github-hashicorp-raft_1.1.2-1_amd64.buildinfo Files: d2b1b6458f45e5e0d928d454aeb4deeb 90036 devel optional golang-github-hashicorp-raft-dev_1.1.2-1_all.deb d91c0915ef228615566a56f797c50c8c 8301 devel optional golang-github-hashicorp-raft_1.1.2-1_amd64.buildinfo Mon Jan 8 10:39:28 UTC 2024 I: diffoscope 240 will be used to compare the two builds: # Profiling output for: /usr/bin/diffoscope --timeout 7200 --html /srv/reproducible-results/rbuild-debian/r-b-build.5KHihXX9/golang-github-hashicorp-raft_1.1.2-1.diffoscope.html --text /srv/reproducible-results/rbuild-debian/r-b-build.5KHihXX9/golang-github-hashicorp-raft_1.1.2-1.diffoscope.txt --json /srv/reproducible-results/rbuild-debian/r-b-build.5KHihXX9/golang-github-hashicorp-raft_1.1.2-1.diffoscope.json --profile=- /srv/reproducible-results/rbuild-debian/r-b-build.5KHihXX9/b1/golang-github-hashicorp-raft_1.1.2-1_amd64.changes /srv/reproducible-results/rbuild-debian/r-b-build.5KHihXX9/b2/golang-github-hashicorp-raft_1.1.2-1_amd64.changes ## command (total time: 0.000s) 0.000s 1 call cmp (internal) ## has_same_content_as (total time: 0.000s) 0.000s 1 call abc.DotChangesFile ## main (total time: 0.638s) 0.638s 2 calls outputs 0.000s 1 call cleanup ## recognizes (total time: 0.053s) 0.052s 12 calls diffoscope.comparators.binary.FilesystemFile 0.000s 10 calls abc.DotChangesFile ## specialize (total time: 0.000s) 0.000s 1 call specialize Mon Jan 8 10:39:29 UTC 2024 I: diffoscope 240 found no differences in the changes files, and a .buildinfo file also exists. Mon Jan 8 10:39:29 UTC 2024 I: golang-github-hashicorp-raft from bookworm built successfully and reproducibly on amd64. Mon Jan 8 10:39:35 UTC 2024 I: Submitting .buildinfo files to external archives: Mon Jan 8 10:39:35 UTC 2024 I: Submitting 12K b1/golang-github-hashicorp-raft_1.1.2-1_amd64.buildinfo.asc Mon Jan 8 10:39:38 UTC 2024 I: Submitting 12K b2/golang-github-hashicorp-raft_1.1.2-1_amd64.buildinfo.asc Mon Jan 8 10:39:39 UTC 2024 I: Done submitting .buildinfo files to http://buildinfo.debian.net/api/submit. Mon Jan 8 10:39:39 UTC 2024 I: Done submitting .buildinfo files. Mon Jan 8 10:39:39 UTC 2024 I: Removing signed golang-github-hashicorp-raft_1.1.2-1_amd64.buildinfo.asc files: removed './b1/golang-github-hashicorp-raft_1.1.2-1_amd64.buildinfo.asc' removed './b2/golang-github-hashicorp-raft_1.1.2-1_amd64.buildinfo.asc'