Files
chia-blockchain/chia/plotters/plotters.py
T
f550555c7d Add plotters. (#8497)
* Initial commit add plotters.

* Lint.

* Add progress for bladebit.

* Address some review comments.

* Oops...

* Add install option.

* Change chiapos to fit the old standard.

* Update chia/plotters/bladebit.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Update chia/plotters/install_plotter.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Lint.

* Remove farmerkey as required.

* Chia plotters chiapos works with no arguments.

* Added get_plotters RPC to support the GUI (#8530)

* Added 'get_plotters' daemon RPC. Probes installed/available plotters on behalf of the GUI.

* Linter fix

* Minor type tweak

* Tweaks.

* Run with default arguments all plotters.

* Fix bug.

* Update chia/plotters/bladebit.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Change bladebit repo.

* Update chia/plotters/bladebit.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Update chia/plotters/plotters.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Update chia/plotters/plotters.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Update chia/plotters/plotters.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Update chia/plotters/plotters.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Update chia/plotters/plotters.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Update chia/plotters/plotters.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Update chia/plotters/plotters.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Update chia/plotters/plotters.py

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>

* Re-added the connect-to-daemon hidden option

* Use connect_to_daemon value for madmax and bladebit plot key resolution.

* Updated --tmp_dir, --tmp_dir2, --final_dir options to match old options from chia plots create

* Add CONNECT_TO_DAEMON as a valid option for madmax/bladebit

* Thread multiplier should be an int

* Passing params for madmax to start_plotting. Still needs cleanup/refactoring.

* Update chia/plotters/bladebit.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Update chia/plotters/madmax.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Filename option -z.

* Factor out calling the plotter.

* First attempt refactor install scripts.

* Switch to exec.

* Attempt to fix mypy warning.

* Remove filename.

* Increase RLIMIT_NOFILE for madmax on non-Windows platforms

* Add trailing path separator to madmax tmpdir/tmp2dir/finaldir arguments (required by madmax)

* Fixes to support madmax plotting from the GUI.
Writing output from the plotters now includes a flush to ensure the plotter log (used by the GUI) is updated frequently.

* Handle madmax's tmptoggle option internally when plotting with the GUI

* Formatting and linter fix

* Fixed the -i option for bladebit

* Construct BladeBit plotting options

* Cleanup code for building plotter command line options

* Added a post-processing step after each plotting job completes. Adds the final_dir plot directory as necessary.

* Fix plotter root path

* Reverting prior checkin. Need to figure out how to handle CHIA_ROOT being overridden

* BladeBit support for Windows

* BladeBit's --memory-json option is used to check memory requirements

* Madmax Windows support

* Plotters directory is now under CHIA_ROOT

* Madmax version detection

* BladeBit will default to 0 threads. BladeBit will max-out available threads with this configuration.

* LGTM fixes

* Module definition for chia.plotters to resolve mypy issues with chiapos (package vs chiapos.py)

* Updated BladeBit build script to account for 1.2.0 changes.
Replaced remaining subprocess.run calls with run_command.
Use BladeBit's reported memory requirement instead of hardcoded value.

* Show a disclaimer when using thirdparty plotter

* Test adding mac madmax plotter to the installers

* Get latest madmax from the latest GH release

* Fix bad var name

* m1 madmax

* Add linux/linux arm

* pip install -e for arm installer, so that its consistent with the other platforms when looking for additional files

* madmax + windows

* Get madmax with Invoke-WebRequest

* Use the correct windows slashes

* add madmax to the list of windows binaries

* Check if madmax exists on windows install and move it to site packages if it does

* Make sure windows has .exe extension for madmax

* Update azure to get latest version of madmax from GH releases

* Bladebit for linux/linux arm

* Fix error with binaries.extend

* Bundle bladebit + windows

* Fix download url for bladebit

* Check for bladebit in windows script. move to correct directory if it exists

* Detect and use packaged plotters

* Removed unnecessary import

* Updating the branch to use chiaplotters_gui for installer verification

* Removed a change that was intended for debugging only

* Remove disclaimer

* Updated for new madmax plotter with k33, k34 support.
Updated chia-blockchain-gui submodule

* Fixed typo

* Package the chia_plot_k34 executable

* Boink

* Revert "Boink"

This reverts commit 8d13c07110.

* Additional chia_plot_k34 spots that I missed

* pyinstaller.spec fix for chia_plot_k34

* Windows installer fix for chia_plot_k34.exe

* Restoring chia-blockchain-gui submodule to 047ce16 (as in main)

* Update to chiapos 1.0.6

Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
Co-authored-by: Jeff Cruikshank <jeff@chia.net>
Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
2021-10-28 15:37:46 -07:00

370 lines
11 KiB
Python

import argparse
import binascii
import os
from enum import Enum
from chia.plotters.bladebit import get_bladebit_install_info, plot_bladebit
from chia.plotters.chiapos import get_chiapos_install_info, plot_chia
from chia.plotters.madmax import get_madmax_install_info, plot_madmax
from chia.plotters.install_plotter import install_plotter
from pathlib import Path
from typing import Any, Dict, Optional
class Options(Enum):
TMP_DIR = 1
TMP_DIR2 = 2
FINAL_DIR = 3
K = 4
MEMO = 5
ID = 6
BUFF = 7
NUM_BUCKETS = 8
STRIPE_SIZE = 9
NUM_THREADS = 10
NOBITFIELD = 11
PLOT_COUNT = 12
MADMAX_NUM_BUCKETS_PHRASE3 = 13
MADMAX_WAITFORCOPY = 14
POOLKEY = 15
FARMERKEY = 16
MADMAX_TMPTOGGLE = 17
POOLCONTRACT = 18
MADMAX_RMULTI2 = 19
BLADEBIT_WARMSTART = 20
BLADEBIT_NONUMA = 21
VERBOSE = 22
OVERRIDE_K = 23
ALT_FINGERPRINT = 24
EXCLUDE_FINAL_DIR = 25
CONNECT_TO_DAEMON = 26
chia_plotter = [
Options.TMP_DIR,
Options.TMP_DIR2,
Options.FINAL_DIR,
Options.K,
Options.MEMO,
Options.ID,
Options.BUFF,
Options.NUM_BUCKETS,
Options.STRIPE_SIZE,
Options.NUM_THREADS,
Options.NOBITFIELD,
Options.OVERRIDE_K,
Options.ALT_FINGERPRINT,
Options.POOLCONTRACT,
Options.FARMERKEY,
Options.POOLKEY,
Options.PLOT_COUNT,
Options.EXCLUDE_FINAL_DIR,
Options.CONNECT_TO_DAEMON,
]
madmax_plotter = [
Options.K,
Options.PLOT_COUNT,
Options.NUM_THREADS,
Options.NUM_BUCKETS,
Options.MADMAX_NUM_BUCKETS_PHRASE3,
Options.TMP_DIR,
Options.TMP_DIR2,
Options.FINAL_DIR,
Options.MADMAX_WAITFORCOPY,
Options.POOLKEY,
Options.FARMERKEY,
Options.POOLCONTRACT,
Options.MADMAX_TMPTOGGLE,
Options.MADMAX_RMULTI2,
Options.CONNECT_TO_DAEMON,
]
bladebit_plotter = [
Options.NUM_THREADS,
Options.PLOT_COUNT,
Options.FARMERKEY,
Options.POOLKEY,
Options.POOLCONTRACT,
Options.ID,
Options.BLADEBIT_WARMSTART,
Options.BLADEBIT_NONUMA,
Options.FINAL_DIR,
Options.VERBOSE,
Options.CONNECT_TO_DAEMON,
]
def get_plotters_root_path(root_path: Path) -> Path:
return root_path / "plotters"
def build_parser(subparsers, root_path, option_list, name, plotter_desc):
parser = subparsers.add_parser(name, description=plotter_desc)
for option in option_list:
if option is Options.K:
parser.add_argument(
"-k",
"--size",
type=int,
help="K value.",
default=32,
)
u_default = 0 if name == "chiapos" else 256
if option is Options.NUM_BUCKETS:
parser.add_argument(
"-u",
"--buckets",
type=int,
help="Number of buckets.",
default=u_default,
)
if option is Options.STRIPE_SIZE:
parser.add_argument(
"-s",
"--stripes",
type=int,
help="Stripe size.",
default=0,
)
if option is Options.TMP_DIR:
parser.add_argument(
"-t",
"--tmp_dir",
type=str,
dest="tmpdir",
help="Temporary directory 1.",
default=str(root_path) + "/",
)
if option is Options.TMP_DIR2:
parser.add_argument(
"-2",
"--tmp_dir2",
type=str,
dest="tmpdir2",
help="Temporary directory 2.",
default=str(root_path) + "/",
)
if option is Options.FINAL_DIR:
parser.add_argument(
"-d",
"--final_dir",
type=str,
dest="finaldir",
help="Final directory.",
default=str(root_path) + "/",
)
if option is Options.BUFF:
parser.add_argument(
"-b",
"--buffer",
type=int,
help="Size of the buffer, in MB.",
default=0,
)
r_default = 4 if name == "madmax" else 0
if option is Options.NUM_THREADS:
parser.add_argument(
"-r",
"--threads",
type=int,
help="Num threads.",
default=r_default,
)
if option is Options.NOBITFIELD:
parser.add_argument(
"-e",
"--nobitfield",
action="store_true",
help="Disable bitfield.",
default=False,
)
if option is Options.MEMO:
parser.add_argument(
"-m",
"--memo",
type=binascii.unhexlify,
help="Memo variable.",
)
if option is Options.ID:
parser.add_argument(
"-i",
"--id",
type=binascii.unhexlify,
help="Plot id",
)
if option is Options.PLOT_COUNT:
parser.add_argument(
"-n",
"--count",
type=int,
help="Number of plots to create (default = 1)",
default=1,
)
if option is Options.MADMAX_NUM_BUCKETS_PHRASE3:
parser.add_argument(
"-v",
"--buckets3",
type=int,
help="Number of buckets for phase 3+4 (default = 256)",
default=256,
)
if option is Options.MADMAX_WAITFORCOPY:
parser.add_argument(
"-w",
"--waitforcopy",
action="store_true",
help="Wait for copy to start next plot",
default=False,
)
if option is Options.MADMAX_TMPTOGGLE:
parser.add_argument(
"-G",
"--tmptoggle",
action="store_true",
help="Alternate tmpdir/tmpdir2 (default = false)",
default=False,
)
if option is Options.POOLCONTRACT:
parser.add_argument(
"-c",
"--contract",
type=str,
help="Pool Contract Address (64 chars)",
default="",
)
if option is Options.MADMAX_RMULTI2:
parser.add_argument(
"-K",
"--rmulti2",
type=int,
help="Thread multiplier for P2 (default = 1)",
default=1,
)
if option is Options.POOLKEY:
parser.add_argument(
"-p",
"--pool-key",
type=binascii.unhexlify,
help="Pool Public Key (48 bytes)",
default="",
)
if option is Options.FARMERKEY:
parser.add_argument(
"-f",
"--farmerkey",
type=binascii.unhexlify,
help="Farmer Public Key (48 bytes)",
default="",
)
if option is Options.BLADEBIT_WARMSTART:
parser.add_argument(
"-w",
"--warmstart",
action="store_true",
help="Warm start",
default=False,
)
if option is Options.BLADEBIT_NONUMA:
parser.add_argument(
"-m",
"--nonuma",
action="store_true",
help="Disable numa",
default=False,
)
if option is Options.VERBOSE:
parser.add_argument(
"-v",
"--verbose",
action="store_true",
help="Set verbose",
default=False,
)
if option is Options.OVERRIDE_K:
parser.add_argument(
"--override-k",
dest="override",
action="store_true",
help="Force size smaller than 32",
default=False,
)
if option is Options.ALT_FINGERPRINT:
parser.add_argument(
"-a",
"--alt_fingerprint",
type=int,
default=None,
help="Enter the alternative fingerprint of the key you want to use",
)
if option is Options.EXCLUDE_FINAL_DIR:
parser.add_argument(
"-x",
"--exclude_final_dir",
action="store_true",
help="Skips adding [final dir] to harvester for farming",
default=False,
)
if option is Options.CONNECT_TO_DAEMON:
parser.add_argument(
"-D",
"--connect-to-daemon",
action="store_true",
help=argparse.SUPPRESS,
default=False,
)
def call_plotters(root_path: Path, args):
# Add `plotters` section in CHIA_ROOT.
chia_root_path = root_path
root_path = get_plotters_root_path(root_path)
if not root_path.is_dir():
if os.path.exists(root_path):
try:
os.remove(root_path)
except Exception as e:
print(f"Exception deleting old root path: {type(e)} {e}.")
if not os.path.exists(root_path):
print(f"Creating plotters folder within CHIA_ROOT: {root_path}")
try:
os.mkdir(root_path)
except Exception as e:
print(f"Cannot create plotters root path {root_path} {type(e)} {e}.")
plotters = argparse.ArgumentParser(description="Available options.")
subparsers = plotters.add_subparsers(help="Available options", dest="plotter")
build_parser(subparsers, root_path, chia_plotter, "chiapos", "Chiapos Plotter")
build_parser(subparsers, root_path, madmax_plotter, "madmax", "Madmax Plotter")
build_parser(subparsers, root_path, bladebit_plotter, "bladebit", "Bladebit Plotter")
install_parser = subparsers.add_parser("install", description="Install custom plotters.")
install_parser.add_argument(
"install_plotter", type=str, help="The plotters available for installing. Choose from madmax or bladebit."
)
args = plotters.parse_args(args)
if args.plotter == "chiapos":
plot_chia(args, chia_root_path)
if args.plotter == "madmax":
plot_madmax(args, chia_root_path, root_path)
if args.plotter == "bladebit":
plot_bladebit(args, chia_root_path, root_path)
if args.plotter == "install":
install_plotter(args.install_plotter, root_path)
def get_available_plotters(root_path) -> Dict[str, Any]:
plotters_root_path: Path = get_plotters_root_path(root_path)
plotters: Dict[str, Any] = {}
chiapos: Optional[Dict[str, Any]] = get_chiapos_install_info()
bladebit: Optional[Dict[str, Any]] = get_bladebit_install_info(plotters_root_path)
madmax: Optional[Dict[str, Any]] = get_madmax_install_info(plotters_root_path)
if chiapos is not None:
plotters["chiapos"] = chiapos
if bladebit is not None:
plotters["bladebit"] = bladebit
if madmax is not None:
plotters["madmax"] = madmax
return plotters